System and method of capturing and generating panoramic three-dimensional images

ABSTRACT

An apparatus comprising a housing, a mount configured to be coupled to a motor to horizontally move the apparatus, a wide-angle lens coupled to the housing, the wide-angle lens being positioned above the mount thereby being along an axis of rotation, the axis of rotation being the axis along which the apparatus rotates, an image capture device within the housing, the image capture device configured to receive two-dimensional images through the wide-angle lens of environment, and a LiDAR device within the housing, the LiDAR device configured to generate depth data based on the environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/806,447, filed Jun. 10, 2022, entitled “System and Method ofCapturing and Generating Panoramic Three-Dimensional Images,” which is acontinuation of U.S. application Ser. No. 17/664,602, filed May 23,2022, entitled “System and Method of Capturing and Generating PanoramicThree-Dimensional Images,” which is a continuation of U.S. applicationSer. No. 17/137,958, filed Dec. 30, 2020, entitled “System and Method ofCapturing and Generating Panoramic Three-Dimensional Images,” and claimspriority to U.S. Application No. 62/955,414, filed Dec. 30, 2019,entitled “System and Method of Capturing and Generating PanoramicThree-Dimensional Images,” all of which are incorporated herein byreference.

BACKGROUND 1. Field of the Invention(s)

Embodiments of the present invention(s) are generally related tocapturing and stitching panoramic images of scenes in a physicalenvironment.

2. Related Art

The popularity of providing three-dimensional (3D) panoramic images ofthe physical world has created many solutions that have the capabilityof capturing multiple two-dimensional (2D) images and creating a 3Dimage based on the captured 2D images. There exist hardware solutionsand software applications (or “apps”) capable of capturing multiple 2Dimages and stitching them into a panoramic image.

Technologies exist for capturing and generating 3D data from a building.However, existing technologies are generally incapable of capturing andgenerating a 3D rendering of an area with bright light. A window withthe sun shining through or an area of a floor or wall with a brightlight usually appears as a hole in the 3D rendering, which may requireadditional post-production work to fill in. This increases theturnaround time and authenticity of the 3D rendering. Furthermore, theoutdoor environment also provides a challenge for many existing 3Dcapture device because structure light may not be utilized to capture 3Dimages.

Other limitations of existing technologies for capturing and generating3D data include the amount of time required to capture and process thedigital images required to produce a 3D panoramic image.

SUMMARY

An example apparatus comprises a housing and a mount configured to becoupled to a motor to horizontally move the apparatus, a wide-angle lenscoupled to the housing, the wide-angle lens being positioned above themount thereby being along an axis of rotation, the axis of rotationbeing the axis along which the apparatus rotates when coupled to themotor, an image capture device within the housing, the image capturedevice configured to receive two-dimensional images through thewide-angle lens of an environment, and a LiDAR device within thehousing, the LiDAR device configured to generate depth data based on theenvironment.

An image capture device may comprise a housing, first motor, awide-angle lens, an image sensor, a mount, a LiDAR, a second motor, anda mirror. The housing may have a front side and a back side. The firstmotor may be coupled to the housing at a first position between thefront side and the back side of the housing, the first motor beingconfigured to horizontally turn the image capture device substantially270 degrees about a vertical axis. The wide-angle lens may be coupled tothe housing at a second position between the front side and the backside of the housing along the vertical axis, the second position being ano-parallax point and the wide-angle lens having a field of view awayfrom the front side of the housing. The image sensor may be coupled tothe housing and configured to generate image signals from light receivedby the wide-angle lens. The mount may be coupled to the first motor. TheLiDAR may be coupled to the housing at a third position, the LiDARconfigured to generate laser pulses and generate depth signals. Thesecond motor may be coupled to the housing. The mirror may be coupled tothe second motor, the second motor may be configured to rotate themirror around a horizontal axis, the mirror including an angled surfaceconfigured to receive the laser pulses from the LiDAR and direct thelaser pulses about the horizontal axis.

In some embodiments, the image sensor is configured to generate a firstplurality of images at different exposures when the image capture deviceis stationary and pointed in a first direction. The first motor may beconfigured to turn the image capture device about the vertical axisafter the first plurality of images are generated. In variousembodiments, the image sensor does not generate images while the firstmotor turns the image capture device and wherein the LiDAR generatesdepth signals based on the laser pulses while the first motor turns theimage capture device. The image sensor may be configured to generate asecond plurality of images at the different exposures when the imagecapture device is stationary and pointed in a second direction and thefirst motor is configured to turn the image capture device 90 degreesabout the vertical axis after the second plurality of images aregenerated. The image sensor may be configured to generate a thirdplurality of images at the different exposures when the image capturedevice is stationary and pointed in a third direction and the firstmotor is configured to turn the image capture device 90 degrees aboutthe vertical axis after the third plurality of images are generated. Theimage sensor may be configured to generate a fourth plurality of imagesat the different exposures when the image capture device is stationaryand pointed in a fourth direction and the first motor is configured toturn the image capture device 90 degrees about the vertical axis afterthe fourth plurality of images are generated.

In some embodiments, the system may further comprise a processorconfigured to blend frames of the first plurality of images before theimage sensor generates the second plurality of images. A remote digitaldevice may be in communication with the image capture device andconfigured to generate a 3D visualization based on the first, second,third, and fourth plurality of images and the depth signals, the remotedigital device being configured to generate the 3D visualization usingno more images than the first, second, third, and fourth plurality ofimages. In some embodiments, the first, second, third, and fourthplurality of images are generated between turns that combined turnsturning the image capture device 270 degrees around the vertical axis.The speed or rotation of the mirror around the horizontal axis increasesas the first motor turns the image capture device. The angled surface ofthe mirror may be 90 degrees. In some embodiments, the LiDAR emits thelaser pulses in a direction that is opposite the front side of thehousing.

An example method comprises receiving light from a wide-angle lens of animage capture device, the wide-angel lens being coupled to a housing ofthe image capture device, the light being received at a field of view ofthe wide-angle lens, the field of view extending away from a front sideof the housing, generating a first plurality of images by an imagesensor of an image capture device using the light from the wide-anglelens, the image sensor being coupled to the housing, the first pluralityof images being at different exposures, horizontally turning the imagecapture device by a first motor substantially 270 degrees about avertical axis, the first motor being coupled to the housing in a firstposition between the front side and a back side of the housing, thewide-angle lens being at a second position along the vertical axis, thesecond position being a no-parallax point, rotating a mirror with anangled surface around horizontal axis by a second motor, the secondmotor being coupled to the housing, generating laser pulses by a LiDAR,the LiDAR being coupled to the housing at a third position, the laserpulse being directed to the rotating mirror while the image capturedevice horizontally turns, and generating depth signals by the LiDARbased on the laser pulses.

Generating the first plurality of images by the image sensor may occurbefore the image captured device horizontally turns. In someembodiments, the image sensor does not generate images while the firstmotor turns the image capture device and wherein the LiDAR generates thedepth signals based on the laser pulses while the first motor turns theimage capture device.

The method may further comprise generating a second plurality of imagesat the different exposures by the image sensor when the image capturedevice is stationary and pointed in a second direction and turning theimage capture device 90 degrees about the vertical axis by the firstmotor after the second plurality of images are generated.

In some embodiments, the method may further comprise generating a thirdplurality of images at the different exposures by the image sensor whenthe image capture device is stationary and pointed in a third directionand turning the image capture device 90 degrees about the vertical axisby the first motor after the third plurality of images are generated.The method may further comprise generating a fourth plurality of imagesat the different exposures by the image sensor when the image capturedevice is stationary and pointed in a fourth direction. The method maycomprise generating a 3D visualization using the first, second, third,and fourth plurality of images and based on the depth signals, thegenerating the 3D visualization not using any other images.

In some embodiments, the method may further comprise blending frames ofthe first plurality of images before the image sensor generates thesecond plurality of images. The first, second, third, and fourthplurality of images may be generated between turns that combined turnsturning the image capture device 270 degrees around the vertical axis.In some embodiments, a speed or rotation of the mirror around thehorizontal axis increases as the first motor turns the image capturedevice.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a depicts a dollhouse view of an example environment, such as ahouse, according to some embodiments.

FIG. 1 b depicts a floorplan view of the first floor of the houseaccording to some embodiments.

FIG. 2 depicts an example eye-level view of the living room which may bepart of a virtual walkthrough.

FIG. 3 depicts one example of an environment capture system according tosome embodiments.

FIG. 4 depicts a rendering of an environment capture system in someembodiments.

FIG. 5 is a depiction of the laser pulses from the LiDAR about theenvironment capture system in some embodiments.

FIG. 6 a depicts a side view of the environment capture system.

FIG. 6 b depicts a view from above the environment capture system insome embodiments.

FIG. 7 depicts a rendering of the components of one example of theenvironment capture system according to some embodiments.

FIG. 8 a depicts example lens dimensions in some embodiments.

FIG. 8 b depicts an example lens design specification in someembodiments.

FIG. 9 a depicts a block diagram of an example of an environment capturesystem according to some embodiments.

FIG. 9 b depicts a block diagram of an example SOM PCBA of theenvironment capture system according to some embodiments.

FIG. 10 a-10 c depicts a process for the environment capture system fortaking images in some embodiments.

FIG. 11 depicts a block diagram of an example environment capable ofcapturing and stitching images to form 3D visualizations according tosome embodiments.

FIG. 12 is a block diagram of an example of the align and stitchingsystem according to some embodiments.

FIG. 13 depicts a flow chart of a 3D panoramic image capture andgeneration process according to some embodiments.

FIG. 14 depicts a flow chart of a 3D and panoramic capture and stitchingprocess according to some embodiments.

FIG. 15 depicts a flow chart showing further detail of one step of the3D and panoramic capture and stitching process of FIG. 14 .

FIG. 16 depicts a block diagram of an example digital device accordingto some embodiments.

DETAILED DESCRIPTION

Many of the innovations described herein are made with reference to thedrawings. Like reference numerals are used to refer to like elements. Inthe following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding. It may be evident, however, that different innovationscan be practiced without these specific details. In other instances,well-known structures and components are shown in block diagram form inorder to facilitate describing the innovations.

Various embodiments of the apparatus provide users with 3D panoramicimages of indoor as well as outdoor environments. In some embodiments,the apparatus may efficiently and quickly provide users with 3Dpanoramic images of indoor and outdoor environments using a single widefield-of-view (FOV) lens and a single light and detection and rangingsensors (LiDAR sensor).

The following is an example use case of an example apparatus describedherein. The following use case is of one of the embodiments. Differentembodiments of the apparatus, as discussed herein, may include one ormore similar features and capabilities as that of the use case.

FIG. 1 a depicts a dollhouse view 100 of an example environment, such asa house, according to some embodiments. The dollhouse view 100 gives anoverall view of the example environment captured by an environmentcapture system (discussed herein). A user may interact with thedollhouse view 100 on a user system by toggling between different viewsof the example environment. For example, the user may interact with area110 to trigger a floorplan view of the first floor of the house, as seenin FIG. 1 b . In some embodiments, the user may interact with icons inthe dollhouse view 100, such as icons 120, 130, and 140, to provide awalkthrough view (e.g., for a 3D walkthrough), a floorplan view, or ameasurement view, respectively.

FIG. 1 b depicts a floorplan view of the first floor of the houseaccording to some embodiments. The floorplan view is a top-down view ofthe first floor of the house. The user may interact with areas of thefloorplan view, such as the area 150, to trigger an eye-level view of aparticular portion of the floorplan, such as a living room. An exampleof the eye-level view of the living room can be found in FIG. 2 whichmay be part of a virtual walkthrough.

The user may interact with a portion of the floorplan 200 correspondingto the area 150 of FIG. 1 b . The user may move a view around the roomas if the user was actually in the living room. In addition to ahorizontal 360° view of the living room, the user may also view ornavigate the floor or ceiling of the living room. Furthermore, the usermay traverse the living room to other parts of the house by interactingwith particular areas of the portion of the floorplan 200, such as areas210 and 220. When the user interacts with the area 220, the environmentcapture system may provide a walking-style transition between the areaof the house substantially corresponding to the region of the housedepicted by area 150 to an area of the house substantially correspondingto the region of the house depicted by the area 220.

FIG. 3 depicts one example of an environment capture system 300according to some embodiments. The environment capture system 300includes lens 310, a housing 320, a mount attachment 330, and a moveablecover 340.

When in use, the environment capture system 300 may be positioned in anenvironment such as a room. The environment capture system 300 may bepositioned on a support (e.g., tripod). The moveable cover 340 may bemoved to reveal a LiDAR and spinnable mirror. Once activated, theenvironment capture system 300 may take a burst of images and then turnusing a motor. The environment capture system 300 may turn on the mountattachment 330. While turning, the LiDAR may take measurements (whileturning, the environment capture system may not take images). Oncedirected to a new direction, the environment capture system may takeanother burst of images before turning to the next direction.

For example, once positioned, a user may command the environment capturesystem 300 to start a sweep. The sweep may be as follows:

-   -   (1) Exposure estimation and then take HDR RGB images        -   Rotate 90 degrees capturing depth data    -   (2) Exposure estimation and then take HDR RGB images        -   Rotate 90 degrees capturing depth data    -   (3) Exposure estimation and then take HDR RGB images        -   Rotate 90 degrees capturing depth data    -   (4) Exposure estimation and then take HDR RGB images        -   Rotate 90 degrees (total 360) capturing depth data

For each burst, there may be any number of images at differentexposures. The environment capture system may blend any number of theimages of a burst together while waiting for another frame and/orwaiting for the next burst.

The housing 320 may protect the electronic components of the environmentcapture system 300 and may provide an interface for user interaction,with a power button, a scan button, and others. For example, the housing320 may include the moveable cover 340, which may be moveable to uncoverthe LiDAR. Furthermore, the housing 320 may include electronicinterfaces, such as a power adapter and indicator lights. In someembodiments, the housing 320 is a molded plastic housing. In variousembodiments, the housing 320 is a combination of one or more of plastic,metal, and polymer.

The lens 310 may be a part of a lens assembly. Further details of thelens assembly may be described in the description of FIG. 7 . The lens310 is strategically placed at a center of an axis of rotation 305 ofthe environment capture system 300. In this example, the axis ofrotation 305 is on the x-y plane. By placing the lens 310 at the centerof the axis of rotation 305, a parallax effect may be eliminated orreduced. Parallax is an error that arises due to the rotation of theimage capture device about a point that is not a non-parallax point(NPP). In this example, the NPP can be found in the center of the 1′ns'sentrance pupil.

For example, assuming that a panoramic image of the physical environmentis generated based on four images captured by the environment capturesystem 300 with a 25% overlap between images of the panoramic image. Ifthere is no parallax, then 25% of one image may overlap exactly withanother image of the same area of the physical environment. Eliminatingor reducing the parallax effect of the multiple images captured by animage sensor through the lens 310 may aid in stitching multiple imagesinto a 2D panoramic image.

The lens 310 may include a large field of view (e.g., lens 310 may be afisheye lens). In some embodiments, the lens may have a horizontal FOV(HFOV) of at least 148 degrees and a vertical FOV (VFOV) of at least 94degrees.

The mount attachment 330 may allow the environment capture system 300 tobe attached to a mount. The mount may allow for the environment capturesystem 300 to be coupled with a tripod, flat surface, or motorized mount(e.g., to move the environment capture system 300). In some embodiments,the mount may allow the environment capture system 300 to rotate along ahorizontal axis.

In some embodiments, the environment capture system 300 may include amotor for turning the environment capture system 300 horizontally aboutthe mount attachment 330.

In some embodiments, a motorized mount may move the environment capturesystem 300 along a horizontal axis, vertical axis, or both. In someembodiments, the motorized mount may rotate or move in the x-y plane.The use of a mount attachment 330 may allow for the environment capturesystem 300 to be coupled to a motorized mount, tripod, or the like tostabilize the environment capture system 300 to reduce or minimizeshaking. In another example, the mount attachment 330 may be coupled toa motorized mount that allows the 3D, and environment capture system 300to rotate at a steady, known speed, which aids the LiDAR in determinethe (x, y, z) coordinates of each laser pulse of the LiDAR.

FIG. 4 depicts a rendering of a environment capture system 400 in someembodiments. The rendering shows the environment capture system 400(which may be an example of the environment capture system 300 of FIG. 3) from a variety of views, such as a front view 410, a top view 420, aside view 430, and a back view 440. In these renderings, the environmentcapture system 400 may include an optional hollow portion depicted inthe side view 430.

In some embodiments, the environment capture system 400 has a width of75 mm, a height of 180 mm, and a depth of 189 mm. It will be appreciatedthat the environment capture system 400 may have any width, height, ordepth. In various embodiments, the ratio of width to height to depth inthe first example is maintained regardless of the specific measurements.

The housing of the 3D and 4 environment capture system 400 may protectthe electronic components of the environment capture system 400 and mayprovide an interface (e.g., screen on back view 440) for userinteraction. Furthermore, the housing may include electronic interfaces,such as a power adapter and indicator lights. In some embodiments, thehousing is a molded plastic housing. In various embodiments, the housingis a combination of one or more of plastic, metal, and polymer. Theenvironment capture system 400 may include a moveable cover, which maybe moveable to uncover the LiDAR and protect the LiDAR from the elementswhen not in use.

The lens depicted on the front view 410 may be a part of a lensassembly. Like the environment capture system 300, the lens of theenvironment capture system 400 is strategically placed at a center of anaxis of rotation. The lens may include a large field of view. In variousembodiments, the lens depicted on the front view 410 is recessed and thehousing is flared such that the wide-angel lens is directly at theno-parallax point (e.g., directly above a mid-point of the mount and/ormotor) but still may take images without interference from the housing.

The mount attachment at the base of the environment capture system 400may allow the environment capture system to be attached to a mount. Themount may allow for the environment capture system 400 to be coupledwith a tripod, flat surface, or motorized mount (e.g., to move theenvironment capture system 400). in some embodiments, the mount may becoupled to an internal motor for turning the environment capture system400 about the mount.

In some embodiments, the mount may allow the environment capture system400 to rotate along a horizontal axis. In various embodiments, amotorized mount may move the environment capture system 400 along ahorizontal axis, vertical axis, or both. The use of a mount attachmentmay allow for the environment capture system 400 to be coupled to amotorized mount, tripod, or the like to stabilize the environmentcapture system 400 to reduce or minimize shaking. In another example,the mount attachment may be coupled to a motorized mount that allows theenvironment capture system 400 to rotate at a steady, known speed, whichaids the LiDAR in determining the (x, y, z) coordinates of each laserpulse of the LiDAR.

In view 430, a mirror 450 is revealed. A LiDAR may emit a laser pulse tothe mirror (in a direction that is opposite the lens view). The laserpulse may hit the mirror 450 which may be angled (e.g., at a 90 degreeangle) The mirror 450 may be coupled to an internal motor that turns themirror such at the laser pulses of the LiDAR may be emitted and/orreceived at many different angles around the environment capture system400.

FIG. 5 is a depiction of the laser pulses from the LiDAR about theenvironment capture system 400 in some embodiments. In this example, thelaser pulses are emitted at the spinning mirror 450. The laser pulsesmay be emitted and received perpendicular to a horizontal axis 602 (seeFIG. 6 ) of the environment capture system 400. The mirror 450 may beangled such that laser pulses from the LiDAR are directed away from theenvironment capture system 400. In some examples, the angle of theangled surface of the mirror may be 90 degrees or be at or between 60degree to 120 degrees.

In some embodiments, while the environment capture system 400 isstationary and in operation, the environment capture system 400 may takea burst of images through the lens. The environment capture system 400may turn on a horizontal motor between bursts of images. While turningalong the mount, the LiDAR of the environment capture system 400 mayemit and/or receive laser pulses which hit the spinning mirror 450. TheLiDAR may generate depth signals from the received laser pulsereflections and/or generate depth data.

In some embodiments, the depth data may be associated with coordinatesabout the environment capture system 400. Similarly, pixels or parts ofimages may be associated with the coordinates about the environmentcapture system 400 to enable the creation of the 3D visualization (e.g.,an image from different directions, a 3D walkthrough, or the like) to begenerated using the images and the depth data.

As shown in FIG. 5 , the LiDAR pulses may be blocked by the bottomportion of the environment capture system 400. It will be appreciatedthat the mirror 450 may spin consistently while the environment capturesystem 400 moves about the mount or the mirror 450 may spin more slowlywhen the environment capture system 400 starts to move and again whenthe environment capture system 400 slows to stop (e.g., maintaining aconstant speed between the starting and stopping of the mount motor).

The LiDAR may receive depth data from the pulses. Due to movement of theenvironment capture system 400 and/or the increase or decrease of thespeed of the mirror 450, the density of depth data about the environmentcapture system 400 may be inconsistent (e.g., more dense in some areasand less dense in others).

FIG. 6 a depicts a side view of the environment capture system 400. Inthis view, the mirror 450 is depicted and may spin about a horizontalaxis. The pulse 604 may be emitted by the LiDAR at the spinning mirror450 and may be emitted perpendicular to the horizontal axis 602.Similarly, the pulse 604 may be received by the LiDAR in a similarmanner.

Although the LiDAR pulses are discussed as being perpendicular to thehorizontal axis 602, it will be appreciated that the LiDAR pulses may beat any angle relative to the horizontal axis 602 (e.g., the mirror anglemay be at any angle including between 60 to 120 degrees). In variousembodiments, the LiDAR emits pulses opposite a front side (e.g., frontside 604) of the environment capture system 400 (e.g., in a directionopposite of the center of the field of view of the lens or towards theback side 606).

As discussed herein, the environment capture system 400 may turn aboutvertical axis 608. In various embodiments, the environment capturesystem 400 takes images and then turns 90 degrees, thereby taking afourth set of images when the environment capture system 400 completesturning 270 degrees from the original starting position where the firstset of images was taken. As such, the environment capture system 400 maygenerate four sets of images between turns totaling 270 degrees (e.g.,assuming that the first set of images was taken before the initialturning of the environment capture system 400). In various embodiments,the images from a single sweep (e.g., the four sets of images) of theenvironment capture system 400 (e.g., taken in a single full rotation ora rotation of 270 degrees about the vertical axis) is sufficient alongwith the depth data acquired during the same sweep to generate the 3Dvisualization without any additional sweeps or turns of the environmentcapture system 400.

It will be appreciated that, in this example, LiDAR pulses are emittedand directed by the spinning mirror in a position that is distant fromthe point of rotation of the environment capture system 400. In thisexample, the distance from the point of rotation of the mount is 608(e.g., the lens may be at the no-parallax point while the lens may be ina position behind the lens relative to the front of the environmentcapture system 400). Since the LiDAR pulses are directed by the mirror450 at a position that is off the point of rotation, the LiDAR may notreceive depth data from a cylinder running from above the environmentcapture system 400 to below the environment capture system 400. In thisexample, the radius of the cylinder (e.g., the cylinder being a lack ofdepth information) may be measured from the center of the point ofrotation of the motor mount to the point where the mirror 450 directsthe LiDAR pulses.

Further, in FIG. 6 b , cavity 610 is depicted. In this example, theenvironment capture system 400 includes the spinning mirror within thebody of the housing of the environment capture system 400. There is acut-out section from the housing. The laser pulses may be reflected bythe mirror out of the housing and then reflections may be received bythe mirror and directed back to the LiDAR to enable the LiDAR to createdepth signals and/or depth data. The base of the body of the environmentcapture system 400 below the cavity 610 may block some of the laserpulses. The cavity 610 ma be defined by the base of the environmentcapture system 400 and the rotating mirror. As depicted in FIG. 6 b ,there may still be a space between an edge of the angled mirror and thehousing of the environment capture system 400 containing the LiDAR.

In various embodiments, the LiDAR is configured to stop emitting laserpulses if the speed of rotation of the mirror drops below a rotatingsafety threshold (e.g., if there is a failure of the motor spinning themirror or the mirror is held in place). In this way, the LiDAR may beconfigured for safety and reduce the possibility that a laser pulse willcontinue to be emitted in the same direction (e.g., at a user's eyes).

FIG. 6 b depicts a view from above the environment capture system 400 insome embodiments. In this example, the front of the environment capturesystem 400 is depicted with the lens recessed and above directly abovethe center of the point of rotation (e.g., above the center of themount). The front of the camera is recessed for the lends and the frontof the housing is flared to allow the field of view of the image sensorto be unobstructed by the housing. The mirror 450 is depicted aspointing upwards.

FIG. 7 depicts a rendering of the components of one example of theenvironment capture system 300 according to some embodiments. Theenvironment capture system 700 includes a front cover 702, a lensassembly 704, a structural frame 706, a LiDAR 708, a front housing 710,a mirror assembly 712, a GPS antenna 714, a rear housing 716, a verticalmotor 718, a display 720 a battery pack 722, a mount 724, and ahorizontal motor 726.

In various embodiments, the environment capture system 700 may beconfigured to scan, align, and create 3D mesh outdoors in full sun aswell as indoors. This removes a barrier to the adoption of other systemswhich are an indoor-only tool. The environment capture system 700 may beable to scan large spaces more quickly than other devices. Theenvironment capture system 700 may, in some embodiments, provide animproved depth accuracy by improving single scan depth accuracy at 90 m.

In some embodiments, the environment capture system 700 may weigh 1 kgor about 1 kg. In one example, the environment capture system 700 mayweigh between 1-3 kg.

The front cover 702, the front housing 710, and the rear housing 716make up a part of the housing. In one example, the front cover may havea width, w, of 75 mm.

The lens assembly 704 may include a camera lens that focuses light ontoan image capture device. The image capture device may capture an imageof a physical environment. The user may place the environment capturesystem 700 to capture one portion of a floor of a building, such as thesecond building 422 of FIG. 1 to obtain a panoramic image of the oneportion of the floor. The environment capture system 700 may be moved toanother portion of the floor of the building to obtain a panoramic imageof another portion of the floor. In one example, the depth of field ofthe image capture device is 0.5 meters to infinity. FIG. 8 a depictsexample lens dimensions in some embodiments.

In some embodiments, the image capture device is a complementarymetal-oxide-semiconductor (CMOS) image sensor (e.g., a Sony IMX283 ˜20Megapixel CMOS MIPI sensor with the NVidia Jetson Nano SOM). In variousembodiments, the image capture device is a charged coupled device (CCD).In one example, the image capture device is a red-green-blue (RGB)sensor. In one embodiment, the image capture device is an infrared (IR)sensor. The lens assembly 704 may be give the image capture device awide field of view.

The image sensor may have many different specifications. In one example,the image sensor includes the following:

Pixels per Column pixels 5496 Pixels per Row pixels 3694 ResolutionMP >20  Image circle diameter mm 15.86 mm Pixel pitch um 2.4 um PixelsPer Degree (PPD) PPD >37  Chief ray angle at full height degree s 3.0°Output Interface — MIPI Green Sensitivity V/lux*s >1.7 SNR (100 lux, 1xgain) dB >65  Dynamic Range dB >70 

Example specifications may be as follows:

F-number — 2.8 Image circle diameter mm 15.86 Minimum object distance mm500 Maximum object distance mm Infinity Chief ray angle at sensor fullheight deg 3.0 L1 diameter mm <60 Total track length (TTL) mm <= 80 BackFocal Length (BFL) mm — Effective Focal Length (EFL) mm — Relativeillumination % >50 Max distortion % <5   52 lp/mm (on-axis) % >85 104lp/mm (on-axis) % >66 208 lp/mm (on-axis) % >45  52 lp/mm (83% field)% >75 104 lp/mm (83% field) % >41 208 lp/mm (83% field) % >25

In various embodiments, in looking at the MTF at FO relative field (ie.,the center), the focus shift may vary from +28 microns at 0.5 m to −25microns at infinity for a total through focus shift of 53 microns.

FIG. 8 b depicts example lens design specifications in some embodiments.

In some examples, the lens assembly 704 has an HFOV of at least 148degrees and a VFOV of at least 94 degrees. In one example, the lensassembly 704 has a field of view of 150°, 180°, or be within a range of145° to 180°. Image capture of a 360° view around the environmentcapture system 700 may be obtained, in one example, with three or fourseparate image captures from the image capture device of environmentcapture system 700. In various embodiments, the image capture device mayhave a resolution of at least 37 pixels per degree. In some embodiments,the environment capture system 700 includes a lens cap (not shown) toprotect the lens assembly 704 when it is not in use. The output of thelens assembly 704 may be a digital image of one area of the physicalenvironment. The images captured by the lens assembly 704 may bestitched together to form a 2D panoramic image of the physicalenvironment. A 3D panoramic may be generated by combining the depth datacaptured by the LiDAR 708 with the 2D panoramic image generated bystitching together multiple images from the lens assembly 704. In someembodiments, the images captured by the environment capture system 402are stitched together by the image processing system 406. In variousembodiments, the environment capture system 402 generates a “preview” or“thumbnail” version of a 2D panoramic image. The preview or thumbnailversion of the 2D panoramic image may be presented on a user system 1110such as an iPad, personal computer, smartphone, or the like. In someembodiments, the environment capture system 402 may generate a mini-mapof a physical environment representing an area of the physicalenvironment. In various embodiments, the image processing system 406generates the mini-map representing the area of the physicalenvironment.

The images captured by the lens assembly 704 may include capture devicelocation data that identifies or indicates a capture location of a 2Dimage. For example, in some implementations, the capture device locationdata can include a global positioning system (GPS) coordinatesassociated with a 2D image. In other implementations, the capture devicelocation data can include position information indicating a relativeposition of the capture device (e.g., the camera and/or a 3D sensor) toits environment, such as a relative or calibrated position of thecapture device to an object in the environment, another camera in theenvironment, another device in the environment, or the like. In someimplementations, this type of location data can be determined by thecapture device (e.g., the camera and/or a device operatively coupled tothe camera comprising positioning hardware and/or software) inassociation with the capture of an image and received with the image.The placement of the lens assembly 704 is not solely by design. Byplacing the lens assembly 704 at the center, or substantially at thecenter, of the axis of rotation, the parallax effect may be reduced.

In some embodiments, the structural frame 706 holds the lens assembly704 and the LiDAR 708 in a particular position and may help protect thecomponents of the example of the environment capture system. Thestructural frame 706 may serve to aid in rigidly mounting the LiDAR 708and place the LiDAR 708 in a fixed position. Furthermore, the fixedposition of the lens assembly 704 and the LiDAR 708 enable a fixedrelationship to align the depth data with the image information toassist with creating the 3D images. The 2D image data and depth datacaptured in the physical environment can be aligned relative to a common3D coordinate space to generate a 3D model of the physical environment.

In various embodiments, the LiDAR 708 captures depth information of aphysical environment. When the user places the environment capturesystem 700 in one portion of a floor of the second building, the LiDAR708 may obtain depth information of objects. The LiDAR 708 may includean optical sensing module that can measure the distance to a target orobjects in a scene by utilizing pulses from a laser to irradiate atarget or scene and measure the time it takes photons to travel to thetarget and return to the LiDAR 708. The measurement may then betransformed into a grid coordinate system by using information derivedfrom a horizontal drive train of the environment capture system 700.

In some embodiments, the LiDAR 708 may return depth data points every 10useconds with a timestamp (of an internal clock). The LiDAR 708 maysample a partial sphere (small holes at top and bottom) every 0.25degrees. In some embodiments, with a data point every 10 usec and 0.25degrees, there may be a 14.40 milliseconds per “disk” of points and 1440disks to make a sphere that is nominally 20.7 seconds. Because each diskcaptures forward and back, the sphere could be captured in a 180° sweep.

In one example, the LiDAR 708 specification may be as follows:

Range (10% reflectance) m 90 Range (20% reflectance) m 130 Range (100%reflectance) m 260 cm 2 Range Precision (1σ @20 m) Wavelength nm 905Laser Safety — Class 1 Point Rate points/s 100,000 Beam Divergencedegrees  0.28 × 0.03 Angular Resolution deg 0.1 Collimated BeamDimensions mm 14.71 × 8.46 (@ 10 cm) Operating Temperature deg C −20 to65 Power (normal mode, active) W 4.83 Power (normal mode, idle) W 4.38Power (standby mode) W 4.07 Time to Active from Off s 3.898 Time toActive from Standby s 0.289 Time to Active from Normal Idle s 0.003Voltage V 10-15.6 Data synchronization — Pulse Per Second (PPS)Dimensions mm 60 × 58 × 56 Weight g 230 Data Latency ms 2 False AlarmRate (@ 100 klx) % <0.01%

One advantage of utilizing LiDAR is that with a LiDAR at the lowerwavelength (e.g., 905 nm, 900-940 nm, or the like) it may allow theenvironment capture system 700 to determine depth information for anoutdoor environment or an indoor environment with bright light.

The placement of the lens assembly 704 and the LiDAR 708 may allow theenvironment capture system 700 or a digital device in communication withthe environment capture system 700 to generate a 3D panoramic imageusing the depth data from the LiDAR 708 and the lens assembly 704. Insome embodiments, the 2D and 3D panoramic images are not generated onthe environment capture system 402.

The output of the LiDAR 708 may include attributes associated with eachlaser pulse sent by the LiDAR 708. The attributes include the intensityof the laser pulse, number of returns, the current return number,classification point, RGC values, GPS time, scan angle, the scandirection, or any combination therein. The depth of field may be (0.5 m;infinity), (1 m; infinity), or the like. In some embodiments, the depthof field is 0.2 m to 1 m and infinity.

In some embodiments, the environment capture system 700 captures fourseparate RBG images using the lens assembly 704 while the environmentcapture system 700 is stationary. In various embodiments, the LiDAR 708captures depth data in four different instances while the environmentcapture system 700 is in motion, moving from one RBG image captureposition to another RBG image capture position. In one example, the 3Dpanoramic image is captured with a 360° rotation of the environmentcapture system 700, which may be called a sweep. In various embodiments,the 3D panoramic image is captured with a less than 360° rotation of theenvironment capture system 700. The output of the sweep may be a sweeplist (SWL), which includes image data from the lens assembly 704 anddepth data from the LiDAR 708 and properties of the sweep, including theGPS location and a timestamp of when the sweep took place. In variousembodiments, a single sweep (e.g., a single 360 degree turn of theenvironment capture system 700) captures sufficient image and depthinformation to generate a 3D visualization (e.g., by the digital devicein communication with the environment capture system 700 that receivesthe imagery and depth data from the environment capture system 700 andcreates the 3D visualization using only the imagery and depth data fromthe environment capture system 700 captured in the single sweep).

In some embodiments, the images captured by the environment capturesystem 402 may be blended, stitched together, and combined with thedepth data from the LiDAR 708 by an image stitching and processingsystem discussed herein.

In various embodiments, the environment capture system 402 and/or anapplication on the user system 1110 may generate a preview or thumbnailversion of a 3D panoramic image. The preview or thumbnail version of the3D panoramic image may be presented on the user system 1110 and may havea lower image resolution than the 3D panoramic image generated by theimage processing system 406. After the lens assembly 704 and the LiDAR708 captures the images and depth data of the physical environment, theenvironment capture system 402 may generate a mini-map representing anarea of the physical environment that has been captured by theenvironment capture system 402. In some embodiments, the imageprocessing system 406 generates the mini-map representing the area ofthe physical environment. After capturing images and depth data of aliving room of a home using the environment capture system 402, theenvironment capture system 402 may generate a top-down view of thephysical environment. A user may use this information to determine areasof the physical environment in which the user has not captured orgenerated 3D panoramic images.

In one embodiment, the environment capture system 700 may interleaveimage capture with the image capture device of the lens assembly 704with depth information capture with the LiDAR 708. For example, theimage capture device may capture an image of section 1605, as seen inFIG. 16 , of the physical environment with the image capture device, andthen LiDAR 708 obtains depth information from section 1605. Once theLiDAR 708 obtains depth information from section 1605, the image capturedevice may move on to capture an image of another section 1610, and thenLiDAR 708 obtains depth information from section 1610, therebyinterleaving image capture and depth information capture.

In some embodiments, the LiDAR 708 may have a field of view of at least145°, depth information of all objects in a 360° view of the environmentcapture system 700 may be obtained by the environment capture system 700in three or four scans. In another example, the LiDAR 708 may have afield of view of at least 150°, 180°, or between 145° to 180°.

An increase in the field of view of the lens reduces the amount of timerequired to obtain visual and depth information of the physicalenvironment around the environment capture system 700. In variousembodiments, the LiDAR 708 has a minimum depth range of 0.5 m. In oneembodiment, the LiDAR 708 has a maximum depth range of greater than 8meters.

The LiDAR 708 may utilize the mirror assembly 712 to direct the laser indifferent scan angles. In one embodiment, the optional vertical motor718 has the capability to move the mirror assembly 712 vertically. Insome embodiments, the mirror assembly 712 may be a dielectric mirrorwith a hydrophobic coating or layer. The mirror assembly 712 may becoupled to the vertical motor 718 that rotates the mirror assembly 712when in use.

The mirror of the mirror assembly 712 may, for example, include thefollowing specifications:

Reflectivity @ 905 nm % >99  Absorption at Visible Wavelengths (380-700nm)** % >60% Clear Aperture % >=85  Laser Damage Threshold @ 905 nmuJ >=0.45 Angle of Incidence (AOI) deg 45 ± 1The mirror of the mirror assembly 712, may, for example, include thefollowing specification for materials and coatings:

S1L1 material Dielectric S1L2 material Hydrophobic S2L1 material BlackPaint Emulsion Powder suspended in paint Substrate material Schott B270IThe hydrophobic coating of the mirror of the mirror assembly 712, may,for example, include a Contact Angle deg >105.The mirror of the mirror assembly 712 may include the following qualityspecifications:

Scratch/Dig Standard 3 HTS: 80 C., 50 hrs. 3 LTS: −30 C., 1000 hrs. 3THS: 60 C./90% RH, 1000 hrs. 3 TC: −30 to 70 C., 50 cyc (30 min./5min./30 min.) 3 (Solvent Resistance), Side a, 50 wipes with ethanol,alcohol, 300 g (Solvent Resistance), Side b, 10 3 wipes with ethanol,alcohol, 200 g (Abrasion Resistance), Side a, 50 wipes, 300 g (AbrasionResistance), Side b, 10 wipes, 200 g 3 (Durability), Side a, 10 tapepeels (CT-18) (Durability), Side b, 5 tape peels (CT-18) 3 UV Resistance(Outdoor environment 3 simulation, 340 nm, 0.35 W/m^(∧)2/nm irradiance,306 min light at 125 C. BTP 54 min. light and deionized water spray(uncontrolled temp) 6 h dark at 95% RH, 24 C. (air)) SurfaceRoughness >=10 Hydrophobic Contact Angle >=10

The vertical motor may include, for example, the followingspecifications:

Maximum Speed RPM 4000 and 6500 Maximum Acceleration deg/sec^(∧)2 300Durability Cycle 70000 Motor Driver Accuracy 1 revolution time variancestandard deviation of <5μsec

Due to the RGB capture device and the LiDAR 708, the environment capturesystem 700 may capture images outside in bright sunlight or inside withbright lights or sunlight glare from windows. In systems that utilizedifferent devices (e.g., structured light devices), they may not be ableto operate in bright environments, whether inside or outside. Thosedevices are often limited to use only inside and only during dawn orsunset to control light. Otherwise, bright spots in a room createartifacts or “holes” in images that must be filled or corrected. Theenvironment capture system 700, however, may be utilized in brightsunlight both inside and outside. The capture device and the LiDAR 708may be able to capture image and depth data in bright environmentswithout artifacts or holes caused by glare or bright light.

In one embodiment, the GPS antenna 714 receives global positioningsystem (GPS) data. The GPS data may be used to determine the location ofthe environment capture system 700 at any given time.

In various embodiments, the display 720 allows the environment capturesystem 700 to provide a current state of the system, such as updating,warming up, scanning, scanning complete, error, and the like.

The battery pack 722 provides power to the environment capture system700. The battery pack 722 may be removable and rechargeable, therebyallowing a user to put in a fresh battery pack 722 while charging adepleted battery pack. In some embodiments, the battery pack 722 mayallow at least 1000 SWLs or at least 250 SWLs of continuous use beforerecharging. The environment capture system 700 may utilize a USB-C plugfor recharging.

In some embodiments, the mount 724 provides a connector for theenvironment capture system 700 to connect to a platform such as a tripodor mount. The horizontal motor 726 may rotate the environment capturesystem 700 around an x-y plane. In some embodiments, the horizontalmotor 726 may provide information to a grid coordinate system todetermine (x, y, z) coordinates associated with each laser pulse. Invarious embodiments, due to the broad field of view of the lens, thepositioning of the lens around the axis of rotation, and the LiDARdevice, the horizontal motor 726 may enable the environment capturesystem 700 to scan quickly.

The horizontal motor 726 may have the following specifications in oneexample:

Maximum Speed deg/sec 60 Maximum Acceleration deg/sec^(∧)2 300 MaximumTorque Nm 0.5 Angular Position Resolution deg <0.125 to <0.025 AngularPosition Accuracy deg <0.1 Encoder Resolution CPR 4096 Durability Cycle70,000

In various embodiments, the mount 724 may include a quick releaseadapter. The holding torque may be, for example, >2.0 Nm and thedurability of the capture operation may be up to or beyond 70,000cycles.

For example, the environment capture system 700 may enable constructionof a 3D mesh of a standard home with a distance between sweeps greaterthan 8 m. A time to capture, process, and align an indoor sweep may beunder 45 seconds. In one example, a time frame from the start of a sweepcapture to when the user can move the environment capture system 700 maybe less than 15 seconds.

In various embodiments, these components provide the environment capturesystem 700 the ability to align scan positions outdoor as well as indoorand therefore create seamless walk-through experiences between indoorand outdoor (this may be a high priority for hotels, vacation rentals,real estate, construction documentation, CRE, and as-built modeling andverification. The environment capture system 700 may also create an“outdoor dollhouse” or outdoor mini-map. The environment capture system700, as shown herein, may also improve the accuracy of the 3Dreconstruction, mainly from a measurement perspective. For scan density,the ability for the user to tune it may also be a plus. These componentsmay also enable the environment capture system 700 the ability tocapture wide empty spaces (e.g., longer range). In order to generate a3D model of wide empty spaces may require the environment capture systemto scan and capture 3D data and depth data from a greater distance rangethan generating a 3D model of smaller spaces.

In various embodiments, these components enable the environment capturesystem 700 to align SWLs and reconstruct the 3D model in a similar wayfor indoor as well as outdoor use. These components may also enable theenvironment capture system 700 to perform geo-localization of 3D models(which may ease integration to Google street view and help align outdoorpanoramas if needed).

The image capture device of the environment capture system 700 may beable to provide a DSLR-like Image with quality printable at 8.5″×11″ for70° VFOV and an RGB image style.

In some embodiments, the environment capture system 700 may take an RGBimage with the image capture device (e.g., using the wide-angle lens)and then move the lens before taking the next RGB image (for a total offour movements using the motor). While the horizontal motor 726 rotatesthe environment capture system 90 degrees, the LiDAR 708 may capturedepth data. In some embodiments, the LiDAR 708 includes an APD array.

In some embodiments, the image and depth data may then be sent to acapture application (e.g., a device in communication with theenvironment capture system 700, such as a smart device or an imagecapture system on a network). In some embodiments, the environmentcapture system 700 may send the image and depth data to the imageprocessing system 406 for processing and generating the 2D panoramicimage or the 3D panoramic image. In various embodiments, the environmentcapture system 700 may generate a sweep list of the captured RGB imageand the depth data from a 360-degree revolution of the environmentcapture system 700. The sweep list may be sent to the image processingsystem 406 for stitching and aligning. The output of the sweep may be aSWL, which includes image data from the lens assembly 704 and depth datafrom the LiDAR 708 and properties of the sweep, including the GPSlocation and a timestamp of when the sweep took place.

In various embodiments, the LIDAR, vertical mirror, RGB lens, tripodmount, and horizontal drive are rigidly mounted within the housing toallow the housing to be opened without requiring the system to berecalibrated.

FIG. 9 a depicts a block diagram 900 of an example of an environmentcapture system according to some embodiments. The block diagram 900includes a power source 902, a power converter 904, an input/output(I/O) printed circuit board assembly (PCBA), a system on module (SOM)PCBA, a user interface 910, a LiDAR 912, a mirror brushless directcurrent (BLCD) motor 914, a drive train 916, wide FOV (WFOV) lens 918,and an image sensor 920.

The power source 902 may be the battery pack 722 of FIG. 7 . The powersource may be a removable, rechargeable battery, such as a lithium-ionbattery (e.g., 4×18650 Li-Ion cell) capable of providing power to theenvironment capture system.

The power converter 904 may change the voltage level from the powersource 902 to a lower or higher voltage level so that it may be utilizedby the electronic components of the environment capture system. Theenvironment capture system may utilize 4×18650 Li-Ion cells in 4S1Pconfiguration, or four series connections and one parallel connectionconfiguration.

In some embodiments, the I/O PCBA 906 may include elements that provideIMU, Wi-Fi, GPS, Bluetooth, inertial measurement unit (IMU), motordrivers, and microcontrollers. In some embodiments, the I/O PCBA 906includes a microcontroller for controlling the horizontal motor andencoding horizontal motor controls as well as controlling the verticalmotor and encoding vertical motor controls.

The SOM PCBA 908 may include a central processing unit (CPU) and/orgraphics processing unit (GPU), memory, and mobile interface. The SOMPCBA 908 may control the LiDAR 912, the image sensor 920, and the I/OPCBA 906. The SOM PCBA 908 may determine the (x, y, z) coordinatesassociated with each laser pulse of the LiDAR 912 and store thecoordinates in a memory component of the SOM PCBA 908. In someembodiments, the SOM PCBA 908 may store the coordinates in the imageprocessing system of the environment capture system 400. In addition tothe coordinates associated with each laser pulse, the SOM PCBA 908 maydetermine additional attributes associated with each laser pulse,including the intensity of the laser pulse, number of returns, thecurrent return number, classification point, RGC values, GPS time, scanangle, and the scan direction.

In some embodiments, the SOM PCBA 908 include an Nvidia SOM PCBAw/CPU/GPU, DDR, eMMC, Ethernet.

The user interface 910 may include physical buttons or switches withwhich the user may interact with. The buttons or switches may providefunctions such as turn the environment capture system on and off, scan aphysical environment, and others. In some embodiments, the userinterface 910 may include a display such as the display 720 of FIG. 7 .

In some embodiments, the LiDAR 912 captures depth information of thephysical environment. The LiDAR 912 includes an optical sensing modulethat can measure the distance to a target or objects in a scene byirradiating the target or scene with light, using pulses from a laser.The optical sensing module of the LiDAR 912 measures the time it takesphotons to travel to said target or object and return after reflectionto a receiver in the LiDAR 912, thereby giving a distance of the LiDARfrom the target or object. Along with the distance, the SOM PCBA 908 maydetermine the (x, y, z) coordinates associated with each laser pulse.The LiDAR 912 may fit within a width of 58 mm, a height of 55 mm, and adepth of 60 mm.

The LiDAR 912 may include a range (10% reflectance) of 90 m, range (20%reflectance) 130 m, range (100% reflectance) 260 m, a range precision(1σ@ 900 m) of 2 cm, a wavelength 1705 nm, and beam divergence of0.28×0.03 degrees.

The SOM PCBA 908 may determine the coordinates based on the location ofthe drive train 916. In various embodiments, the LiDAR 912 may includeone or more LiDAR devices. Multiple LiDAR devices may be utilized toincrease the LiDAR resolution.

The mirror brushless direct current (BLCD) motor 914 may control themirror assembly 712 of FIG. 7 .

In some embodiments, the drive train 916 may include the horizontalmotor 726 of FIG. 7 . The drive train 916 may provide rotation of theenvironment capture system when it is mounted on a platform such as atripod. The drive train 916 may include a stepper motor Nema 14, worm &plastic wheel drive train, clutch, bushing bearing, and a backlashprevention mechanism. In some embodiments, the environment capturesystem may be able to complete a scan in less than 17 seconds. Invarious embodiments, the drive train 916 has a maximum speed of 60degrees/second, a maximum acceleration of 300 degrees/seconds², amaximum torque of 0.5 nm, an angular position accuracy of less than 0.1degrees, and an encoder resolution of about 4096 counts per revolution.

In some embodiments, the drive train 916 includes a vertical monogonmirror and motor. In this example, the drive train 916 may include aBLDC motor, an external hall effect sensor, a magnet (paired with Halleffect sensor), a mirror bracket, and a mirror. The drive train 916 inthis example may have a maximum speed of 4,000 RPM and a maximumacceleration of 300 degrees/sec{circumflex over ( )}2. In someembodiments, the monogon mirror is a dielectric mirror. In oneembodiment, the monogon mirror includes a hydrophobic coating or layer.

The placement of the components of the environment capture system issuch that the lens assembly and the LiDAR are substantially placed at acenter of an axis of rotation. This may reduce the image parallax thatoccurs when an image capture system is not placed at the center of theaxis of rotation.

In some embodiments, the WFOV lens 918 may be the lens of the lensassembly 704 of FIG. 7 . The WFOV lens 918 focuses light onto an imagecapture device. In some embodiments, the WFOV lens may have a FOV of atleast 145 degrees. With such a wide FOV, an image capture of a360-degree view around the environment capture system may be obtainedwith three separate image captures of the image capture device. In someembodiments, the WFOV lens 918 may be about ˜60 mm diameter and ˜80 mmtotal track length (TTL). In one example, the WFOV lens 918 may includea horizontal field of view that is greater than or equal to 148.3degrees and a vertical field of view that is greater than or equal to 94degrees.

An image capture device may include the WFOV lens 918 and the imagesensor 920. The image sensor 920 may be a CMOS image sensor. In oneembodiment, the image sensor 920 is a charged coupled device (CCD). Insome embodiments, the image sensor 920 is a red-green-blue (RGB) sensor.In one embodiment, the image sensor 920 is an IR sensor. In variousembodiments, the image capture device may have a resolution of at least35 pixels per degree (PPD).

In some embodiments, the image capture device may include an F-number off/2.4, Image circle diameter of 15.86 mm, Pixel pitch of 2.4 um,HFOV>148.3°, VFOV>94.0°, Pixels per degree >38.0 PPD, Chief ray angle atfull height of 3.0°, Minimum object distance 1300 mm, Maximum objectdistance infinity, Relative illumination >130%, Max distortion <90%, andSpectral transmission variation <=5%.

In some embodiments, the lens may include F-number 2.8, Image circlediameter 15.86 mm, Pixels per degree >37, Chief ray angle at sensor fullheight 3.0, L1 diameter <60 mm, TTL<80 mm, and Relative illumination>50%.

The lens may include 52 lp/mm (on-axis)>85%, 104 lp/mm (on-axis)>66%,1308 lp/mm (on-axis)>45%, 52 lp/mm (83% field)>75%, 104 lp/mm (83%field)>41%, and 1308 lp/mm (83% field)>25%.

The environment capture system may have a resolution of >20 MP, greensensitivity >1.7 V/lux*s, SNR (100 lux, lx gain)>65 dB, and a dynamicrange of >70 dB.

FIG. 9 b depicts a block diagram of an example SOM PCBA 908 of theenvironment capture system according to some embodiments. The SOM PCBA908 may include a communication component 922, a LiDAR control component924, a LiDAR location component 926, a user interface component 928, aclassification component 930, a LiDAR datastore 932, and a capturedimage datastore 934.

In some embodiments, the communication component 922 may send andreceive requests or data between any of the components of the SOM PCBA1008 and components of the environment capture system of FIG. 9 a.

In various embodiments, the LiDAR control component 924 may controlvarious aspects of the LiDAR. For example, the LiDAR control component924 may send a control signal to the LiDAR 912 to start sending out alaser pulse. The control signal sent by the LiDAR control component 924may include instructions on the frequency of the laser pulses.

In some embodiments, the LiDAR location component 926 may utilize GPSdata to determine the location of the environment capture system. Invarious embodiments, the LiDAR location component 926 utilizes theposition of the mirror assembly to determine the scan angle and (x, y,z) coordinates associated with each laser pulse. The LiDAR locationcomponent 926 may also utilize the IMU to determine the orientation ofthe environment capture system.

The user interface component 928 may facilitate user interaction withthe environment capture system. In some embodiments, the user interfacecomponent 928 may provide one or more user interface elements with whicha user may interact. The user interface provided by the user interfacecomponent 928 may be sent to the user system 1110. For example, the userinterface component 928 may provide to the user system (e.g., a digitaldevice) a visual representation of an area of a floorplan of a building.As the user places the environment capture system in different parts ofthe story of the building to capture and generate 3D panoramic images,the environment capture system may generate the visual representation ofthe floorplan. The user may place the environment capture system in anarea of the physical environment to capture and generate 3D panoramicimages in that region of the house. Once the 3D panoramic image of thearea has been generated by the image processing system, the userinterface component may update the floorplan view with a top-down viewof the living room area depicted in FIG. 1 b . In some embodiments, thefloorplan view 200 may be generated by the user system 1110 after asecond sweep of the same home, or floor of a building has been captured.

In various embodiments, the classification component 930 may classifythe type of physical environment. The classification component 930 mayanalyze objects in the images or objects in images to classify the typeof physical environment was captured by the environment capture system.In some embodiments, the image processing system may be responsible forclassifying the type of physical environment that was captured by theenvironment capture system 400.

The LiDAR datastore 932 may be any structure and/or structures suitablefor captured LiDAR data (e.g., an active database, a relationaldatabase, a self-referential database, a table, a matrix, an array, aflat file, a documented-oriented storage system, a non-relational No-SQLsystem, an FTS-management system such as Lucene/Solar, and/or the like).The image datastore 408 may store the captured LiDAR data. However, theLiDAR datastore 932 may be utilized to cache the captured LiDAR data incases where the communication network 404 is non-functional. Forexample, in cases where the environment capture system 402 and the usersystem 1110 are in a remote location with no cellular network or in aregion with no Wi-Fi, the LiDAR datastore 932 may store the capturedLiDAR data until they can be transferred to the image datastore 934.

Similar to the LiDAR datastore, the captured image datastore 934 may beany structure and/or structures suitable for captured images (e.g., anactive database, a relational database, a self-referential database, atable, a matrix, an array, a flat file, a documented-oriented storagesystem, a non-relational No-SQL system, an FTS-management system such asLucene/Solar, and/or the like). The image datastore 934 may store thecaptured images.

FIG. 10 a-10 c depicts a process for the environment capture system 400for taking images in some embodiments. As depicted in FIG. 10 a-10 c ,the environment capture system 400 may take a burst of images atdifferent exposures. A burst of images may be a set of images, each withdifferent exposures. The first image burst happens at time 0.0. Theenvironment capture system 400 may receive the first frame and thenassess the frame while waiting for the second frame. FIG. 10 a indicatesthat the first frame is blended before the second frame arrives. In someembodiments, the environment capture system 400 may process each frameto identify pixels, color, and the like. Once the next frame arrives,the environment capture system 400 may process the recently receivedframe and then blend the two frames together.

In various embodiments, the environment capture system 400 performsimage processing to blend the sixth frame and further assess the pixelsin the blended frame (e.g., the frame that may include elements from anynumber of the frames of the image burst). During the last step prior toor during movement (e.g., turning) of the environment capture system400, the environment capture system 400 may optionally transfer theblended image from the graphic processing unit to CPU memory.

The process continues in FIG. 10 b . At the beginning of FIG. 10 b , theenvironment capture system 400 conducts another burst. The environmentcapture system 400 may compress the blended frames and/or all or partsof the captured frames using J×R). Like FIG. 10 a , a burst of imagesmay be a set of images, each with different exposures (the length ofexposure for each frame the set may the same and in the same order asother bursts covered in FIGS. 10 a and 10 c ). The second image bursthappens at time 2 second. The environment capture system 400 may receivethe first frame and then assess the frame while waiting for the secondframe. FIG. 10 b indicates that the first frame is blended before thesecond frame arrives. In some embodiments, the environment capturesystem 400 may process each frame to identify pixels, color, and thelike. Once the next frame arrives, the environment capture system 400may process the recently received frame and then blend the two framestogether.

In various embodiments, the environment capture system 400 performsimage processing to blend the sixth frame and further assess the pixelsin the blended frame (e.g., the frame that may include elements from anynumber of the frames of the image burst). During the last step prior toor during movement (e.g., turning) of the environment capture system400, the environment capture system 400 may optionally transfer theblended image from the graphic processing unit to CPU memory.

After turning, the environment capture system 400 may continue theprocess by conducting another color burst (e.g., after turning 180degrees) at about time 3.5 seconds. The environment capture system 400may compress the blended frames and/or all or parts of the capturedframes using J×R). The burst of images may be a set of images, each withdifferent exposures (the length of exposure for each frame the set maythe same and in the same order as other bursts covered in FIGS. 10 a and10 c ). The environment capture system 400 may receive the first frameand then assess the frame while waiting for the second frame. FIG. 10 bindicates that the first frame is blended before the second framearrives. In some embodiments, the environment capture system 400 mayprocess each frame to identify pixels, color, and the like. Once thenext frame arrives, the environment capture system 400 may process therecently received frame and then blend the two frames together.

In various embodiments, the environment capture system 400 performsimage processing to blend the sixth frame and further assess the pixelsin the blended frame (e.g., the frame that may include elements from anynumber of the frames of the image burst). During the last step prior toor during movement (e.g., turning) of the environment capture system400, the environment capture system 400 may optionally transfer theblended image from the graphic processing unit to CPU memory.

The last burst happens at time 5 seconds in FIG. 10 c . The environmentcapture system 400 may compress the blended frames and/or all or partsof the captured frames using J×R). The burst of images may be a set ofimages, each with different exposures (the length of exposure for eachframe the set may the same and in the same order as other bursts coveredin FIGS. 10 a and 10 b ). The environment capture system 400 may receivethe first frame and then assess the frame while waiting for the secondframe. FIG. 10 c indicates that the first frame is blended before thesecond frame arrives. In some embodiments, the environment capturesystem 400 may process each frame to identify pixels, color, and thelike. Once the next frame arrives, the environment capture system 400may process the recently received frame and then blend the two framestogether.

In various embodiments, the environment capture system 400 performsimage processing to blend the sixth frame and further assess the pixelsin the blended frame (e.g., the frame that may include elements from anynumber of the frames of the image burst). During the last step prior toor during movement (e.g., turning) of the environment capture system400, the environment capture system 400 may optionally transfer theblended image from the graphic processing unit to CPU memory.

The dynamic range of an image capture device is a measure of how muchlight an image sensor can capture. The dynamic range is the differencebetween the darkest area to the brightest area of an image. There aremany ways to increase the dynamic range of the image capture device, oneof which is to capture multiple images of the same physical environmentusing different exposures. An image captured with a short exposure willcapture brighter areas of the physical environment, while a longexposure will capture darker physical environment areas. In someembodiments, the environment capture system may capture multiple imageswith six different exposure times. Some or all of the images captured bythe environment capture system are used to generate 2D images with highdynamic range (HDR). One or more of the captured images may be used forother functions such as ambient light detection, flicker detection, andthe like.

A 3D panoramic image of the physical environment may be generated basedon four separate image captures of the image capture device and fourseparate depth data capture of the LiDAR device of the environmentcapture system. Each of the four separate image captures may include aseries of image captures of different exposure times. A blendingalgorithm may be used to blend the series of image captures with thedifferent exposure times to generate one of four RGB image captures,which may be utilized to generate a 2D panoramic image. For example, theenvironment capture system may be used to capture a 3D panoramic imageof a kitchen. Images of one wall of the kitchen may include a window, animage with an image captured with a shorter exposure may provide theview out the window but may leave the rest of the kitchen underexposed.In contrast, another image captured with a longer exposure may providethe view of the interior of the kitchen. The blending algorithm maygenerate a blended RGB image by blending the view out the window of thekitchen from one image with the rest of the kitchen's view from anotherimage.

In various embodiments, the 3D panoramic image may be generated based onthree separate image captures of the image capture device and fourseparate depth data captures of the LiDAR device of the environmentcapture system. In some embodiments, the number of image captures, andthe number of depth data captures may be the same. In one embodiment,the number of image captures, and the number of depth data captures maybe different.

After capturing a first of a series of images with one exposure time, ablending algorithm receives the first of the series of images, calculateinitial intensity weights for that image, and set that image as abaseline image for combining the subsequently received images. In someembodiments, the blending algorithm may utilize a graphic processingunit (GPU) image processing routine such as a “blend_kernel” routine.The blending algorithm may receive subsequent images that may be blendedwith previously received images. In some embodiments, the blendingalgorithm may utilize a variation of the blend_kernel GPU imageprocessing routine.

In one embodiment, the blending algorithm utilizes other methods ofblending multiple images, such as determining the difference between thedarkest and brightest part, or contrast, of the baseline image todetermine if the baseline image may be overexposed or under-exposed. Forexample, a contrast value less than a predetermine contrast thresholdmeans that the baseline image is overexposed or under-exposed. In oneembodiment, the contrast of the baseline image may be calculated bytaking an average of the image's light intensity or a subset of theimage. In some embodiments, the blending algorithm calculates an averagelight intensity for each row or column of the image. In someembodiments, the blending algorithm may determine a histogram of each ofthe images received from the image capture device and analyze thehistogram to determine light intensities of the pixels which make upeach of the images.

In various embodiments, the blending may involve sampling colors withintwo or more images of the same scene, including along objects and seems.If there is a significant difference in color between the two images(e.g., within a predetermined threshold of color, hue, brightness,saturation, and/or the like), a blending module (e.g., on theenvironment capture system 400 or the user device 1110) may blend apredetermined size of both images along the position where there is thedifference. In some embodiments, the greater the difference in color orimage at a position in the image, the greater the amount of space aroundor near the position may be blended.

In some embodiments, after blending, the blending module (e.g., on theenvironment capture system 400 or the user device 1110) may re-scan andsample colors along the image(s) to determine if there are otherdifferences in image or color that exceed the predetermined threshold ofcolor, hue, brightness, saturation, and/or the like. If so, the blendingmodule may identify the portions within the image(s) and continue toblend that portion of the image. The blending module may continue toresample the images along the seam until there are no further portionsof the images to blend (e.g., any differences in color are below thepredetermined threshold(s).)

FIG. 11 depicts a block diagram of an example environment 1100 capableof capturing and stitching images to form 3D visualizations according tosome embodiments. The example environment 1100 includes 3D and panoramiccapture and stitching system 1102, a communication network 1104, animage stitching and processor system 1106, an image datastore 1108, auser system 1110, and a first scene of a physical environment 1112. The3D and panoramic capture and stitching system 1102 and/or the usersystem 1110 may include an image capture device (e.g., environmentcapture system 400) that may be used to capture images of an environment(e.g., the physical environment 1112).

The 3D and panoramic capture and stitching system 1102 and the imagestitching and processor system 1106 may be a part of the same system(e.g., part of one or more digital devices) that are communicativelycoupled to the environment capture system 400. In some embodiments, oneor more of the functionality of the components of the 3D and panoramiccapture and stitching system 1102 and the image stitching and processorsystem 1106 may be performed by the environment capture system 400.Similarly or alternatively, 3D and panoramic capture and stitchingsystem 1102 and the image stitching and processor system 1106 may beperformed by the user system 1110 and/or the image stitching andprocessor system 1106

The 3D panoramic capture and stitching system 1102 may be utilized by auser to capture multiple 2D images of an environment, such as the insideof a building and/or and outside of the building. For example, the usermay utilize the 3D and panoramic capture and stitching system 1102 tocapture multiple 2D images of the first scene of the physicalenvironment 1112 provided by the environment capture system 400. The 3Dand panoramic capture and stitching system 1102 may include an aligningand stitching system 1114. Alternately, the user system 1110 may includethe aligning and stitching system 1114.

The aligning and stitching system 1114 may be software, hardware, or acombination of both configured to provide guidance to the user of animage capture system (e.g., on the 3D and panoramic capture andstitching system 1102 or the user system 1110) and/or process images toenable improved panoramic pictures to be made (e.g., through stitching,aligning, cropping, and/or the like). The aligning and stitching system1114 may be on a computer-readable media (described herein). In someembodiments, the aligning and stitching system 1114 may include aprocessor for performing functions.

An example of the first scene of the physical environment 1112 may beany room, real estate, or the like (e.g., a representation of a livingroom). In some embodiments, the 3D and panoramic capture and stitchingsystem 1102 is utilized to generate 3D panoramic images of indoorenvironments. The 3D panoramic capture and stitching system 1102 may, insome embodiments, be the environment capture system 400 discussed withregard to FIG. 4 .

In some embodiments, the 3D panoramic capture and stitching system 1102may in communication with a device for capturing images and depth dataas well as software (e.g., the environment capture system 400). All orpart of the software may be installed on the 3D panoramic capture andstitching system 1102, the user system 1110, the environment capturesystem 400, or both. In some embodiments, the user may interact with the3D and panoramic capture and stitching system 1102 via the user system1110.

The 3D and panoramic capture and stitching system 1102 or the usersystem 1110 may obtain multiple 2D images. The 3D and panoramic captureand stitching system 1102 or the user system 1110 may obtain depth data(e.g., from a LiDAR device or the like).

In various embodiments, an application on the user system 1110 (e.g., asmart device of the user such as a smartphone or tablet computer) or anapplication on the environment capture system 400 may provide visual orauditory guidance to the user for taking images with the environmentcapture system 400. Graphical guidance may include, for example, afloating arrow on a display of the environment capture system 400 (e.g.,on a viewfinder or LED screen on the back of the environment capturesystem 400) to guide the user on where to position and/or point an imagecapture device. In another example, the application may provide audioguidance on where to position and/or point the image capture device.

In some embodiments, the guidance may allow the user to capture multipleimages of the physical environment without the help of a stabilizingplatform such as a tripod. In one example, the image capture device maybe a personal device such as a smartphone, tablet, media tablet, laptop,and the like. The application may provide direction on position for eachsweep, to approximate the no-parallax point based on position of theimage capture device, location information from the image capturedevice, and/or previous image of the image capture device.

In some embodiments, the visual and/or auditory guidance enables thecapture of images that can be stitched together to form panoramaswithout a tripod and without camera positioning information (e.g.,indicating a location, position, and/or orientation of the camera from asensor, GPS device, or the like).

The aligning and stitching system 1114 may align or stitch 2D images(e.g., captured by the user system 1110 or the 3D panoramic capture andstitching system 1102) to obtain a 2D panoramic image.

In some embodiments, the aligning and stitching system 1114 utilizes amachine learning algorithm to align or stitch multiple 2D images into a2D panoramic image. The parameters of the machine learning algorithm maybe managed by the aligning and stitching system 1114. For example, the3D and panoramic capture and stitching system 1102 and/or the aligningand stitching system 1114 may recognize objects within the 2D images toaid in aligning the images into a 2D panoramic image.

In some embodiments, the aligning and stitching system 1114 may utilizedepth data and the 2D panoramic image to obtain a 3D panoramic image.The 3D panoramic image may be provided to the 3D and panoramic stitchingsystem 1102 or the user system 1110. In some embodiments, the aligningand stitching system 1114 determines 3D/depth measurements associatedwith recognized objects within a 3D panoramic image and/or sends one ormore 2D images, depth data, 2D panoramic image(s), 3D panoramic image(s)to the image stitching and processor system 106 to obtain a 2D panoramicimage or a 3D panoramic image with pixel resolution that is greater thanthe 2D panoramic image or the 3D panoramic image provided by the 3D andpanoramic capture and stitching system 1102.

The communication network 1104 may represent one or more computernetworks (e.g., LAN, WAN, or the like) or other transmission mediums.The communication network 1104 may provide communication between systems1102, 1106-1110, and/or other systems described herein. In someembodiments, the communication network 104 includes one or more digitaldevices, routers, cables, buses, and/or other network topologies (e.g.,mesh, and the like). In some embodiments, the communication network 1104may be wired and/or wireless. In various embodiments, the communicationnetwork 1104 may include the Internet, one or more wide area networks(WANs) or local area networks (LANs), one or more networks that may bepublic, private, IP-based, non-IP based, and so forth.

The image stitching and processor system 1106 may process 2D imagescaptured by the image capture device (e.g., the environment capturesystem 400 or a user device such as a smartphone, personal computer,media tablet, or the like) and stitch them into a 2D panoramic image.The 2D panoramic image processed by the image stitching and processorsystem 106 may have a higher pixel resolution than the panoramic imageobtained by the 3D and panoramic capture and stitching system 1102.

In some embodiments, the image stitching and processor system 1106receives and processes the 3D panoramic image to create a 3D panoramicimage with pixel resolution that is higher than that of the received 3Dpanoramic image. The higher pixel resolution panoramic images may beprovided to an output device with a higher screen resolution than theuser system 1110, such as a computer screen, projector screen, and thelike. In some embodiments, the higher pixel resolution panoramic imagesmay provide to the output device a panoramic image in greater detail andmay be magnified.

The image datastore 1108 may be any structure and/or structures suitablefor captured images and/or depth data (e.g., an active database, arelational database, a self-referential database, a table, a matrix, anarray, a flat file, a documented-oriented storage system, anon-relational No-SQL system, an FTS-management system such asLucene/Solar, and/or the like). The image datastore 1108 may storeimages captured by the image capture device of the user system 1110. Invarious embodiments, the image datastore 1108 stores depth data capturedby one or more depth sensors of the user system 1110. In variousembodiments, the image datastore 1108 stores properties associated withthe image capture device or properties associated with each of themultiple image captures or depth captures used to determine the 2D or 3Dpanoramic image. In some embodiments, the image datastore 1108 storespanoramic 2D or 3D panoramic images. The 2D or 3D panoramic images maybe determined by the 3D and panoramic capture and stitching system 1102or the image stitching and processor system 106.

The user system 1110 may communicate between users and other associatedsystems. In some embodiments, the user system 1110 may be or include oneor more mobile devices (e.g., smartphones, cell phones, smartwatches, orthe like).

The user system 1110 may include one or more image capture devices. Theone or more image capture devices can include, for example, RGB cameras,HDR cameras, video cameras, IR cameras, and the like.

The 3D and panoramic capture and stitching system 1102 and/or the usersystem 1110 may include two or more capture devices may be arranged inrelative positions to one another on or within the same mobile housingsuch that their collective fields of view span up to 360°. In someembodiments, pairs of image capture devices can be used capable ofgenerating stereo-image pairs (e.g., with slightly offset yet partiallyoverlapping fields of view). The user system 1110 may include two imagecapture devices with vertical stereo offset fields-of-view capable ofcapturing vertical stereo image pairs. In another example, the usersystem 1110 can comprise two image capture devices with vertical stereooffset fields-of-view capable of capturing vertical stereo image pairs.

In some embodiments, the user system 1110, environment capture system400, or the 3D and panoramic capture and stitching system 1102 maygenerate and/or provide image capture position and location information.For example, the user system 1110 or the 3D and panoramic capture andstitching system 1102 may include an inertial measurement unit (IMU) toassist in determining position data in association with one or moreimage capture devices that capture the multiple 2D images. The usersystem 1110 may include a global positioning sensor (GPS) to provide GPScoordinate information in association with the multiple 2D imagescaptured by one or more image capture devices.

In some embodiments, users may interact with the aligning and stitchingsystem 1114 using a mobile application installed in the user system1110. The 3D and panoramic capture and stitching system 1102 may provideimages to the user system 1110. A user may utilize the aligning andstitching system 1114 on the user system 1110 to view images andpreviews.

In various embodiments, the aligning and stitching system 1114 may beconfigured to provide or receive one or more 3D panoramic images fromthe 3D and panoramic capture and stitching system 1102 and/or the imagestitching and processor system 1106. In some embodiments, the 3D andpanoramic capture and stitching system 1102 may provide a visualrepresentation of a portion of a floorplan of a building, which has beencaptured by the 3D and panoramic capture and stitching system 1102 tothe user system 1110.

The user of the system 1110 may navigate the space around the area andview different rooms of the house. In some embodiments, the user of theuser system 1110 may display the 3D panoramic images, such as theexample 3D panoramic image, as the image stitching and processor system1106 completes the generation of the 3D panoramic image. In variousembodiments, the user system 1110 generates a preview or thumbnail ofthe 3D panoramic image. The preview 3D panoramic image may have an imageresolution that is lower than a 3D panoramic image generated by the 3Dand panoramic capture and stitching system 1102.

FIG. 12 is a block diagram of an example of the align and stitchingsystem 1114 according to some embodiments. The align and stitchingsystem 1114 includes a communication module 1202, an image captureposition module 1204, a stitching module 1206, a cropping module 1208, agraphical cut module 1210, a blending module 1211, a 3D image generator1214, a captured 2D image datastore 1216, a 3D panoramic image datastore1218, and a guidance module 220. It may be appreciated that there may beany number of modules of the aligning and stitching system 1114 thatperform one or more different functions as described herein.

In some embodiments, the aligning and stitching system 1114 includes animage capture module configured to receive images from one or more imagecapture devices (e.g., cameras). The aligning and stitching system 1114may also include a depth module configured to receive depth data from adepth device such as a LiDAR if available.

The communication module 1202 may send and receive requests, images, ordata between any of the modules or datastores of the aligning andstitching system 1114 and components of the example environment 1100 ofFIG. 11 . Similarly, the aligning and stitching system 1114 may send andreceive requests, images, or data across the communication network 1104to any device or system.

In some embodiments, the image capture position module 1204 maydetermine image capture device position data of an image capture device(e.g., a camera which may be a stand-alone camera, smartphone, mediatablet, laptop, or the like). Image capture device position data mayindicate a position and orientation of an image capture device and/orlens. In one example, the image capture position module 1204 may utilizethe IMU of the user system 1110, camera, digital device with a camera,or the 3D and panoramic capture and stitching system 1102 to generateposition data of the image capture device. The image capture positionmodule 1204 may determine the current direction, angle, or tilt of oneor more image capture devices (or lenses). The image capture positionmodule 1204 may also utilize the GPS of the user system 1110 or the 3Dand panoramic capture and stitching system 1102.

For example, when a user wants to use the user system 1110 to capture a360° view of the physical environment, such as a living room, the usermay hold the user system 1110 in front of them at eye level to start tocapture one of a multiple of images which will eventually become a 3Dpanoramic image. To reduce the amount of parallax to the image andcapture images better suited for stitching and generating 3D panoramicimages, it may be preferable if one or more image capture devices rotateat the center of the axis of rotation. The aligning and stitching system1114 may receive position information (e.g., from the IMU) to determinethe position of the image capture device or lens. The aligning andstitching system 1114 may receive and store a field of view of the lens.The guidance module 1220 may provide visual and/or audio informationregarding a recommended initial position of the image capture device.The guidance module 1220 may make recommendations for positioning theimage capture device for subsequent images. In one example, the guidancemodule 1220 may provide guidance to the user to rotate and position theimage capture device such that the image capture device rotates close toa center of rotation. Further, the guidance module 1220 may provideguidance to the user to rotate and position the image capture devicesuch that subsequent images are substantially aligned based oncharacteristics of the field of view and/or image capture device.

The guidance module 1220 may provide the user with visual guidance. Forexample, the guidance module 1220 may place markers or an arrow in aviewer or display on the user system 1110 or the 3D and panoramiccapture and stitching system 1102. In some embodiments, the user system1110 may be a smartphone or tablet computer with a display. When takingone or more pictures, the guidance module 1220 may position one or moremarkers (e.g., different color markers or the same markers) on an outputdevice and/or in a viewfinder. The user may then use the markers on theoutput device and/or viewfinder to align the next image.

There are numerous techniques for guiding the user of the user system1110 or the 3D and panoramic capture and stitching system 1102 to takemultiple images for ease of stitching the images into a panorama. Whentaking a panorama from multiple images, images may be stitched together.To improve time, efficiency, and effectiveness of stitching the imagestogether with reduced need of correcting artifacts or misalignments, theimage capture position module 1204 and the guidance module 1220 mayassist the user in taking multiple images in positions that improve thequality, time efficiency, and effectiveness of image stitching for thedesired panorama.

For example, after taking the first picture, the display of the usersystem 1110 may include two or more objects, such as circles. Twocircles may appear to be stationary relative to the environment and twocircles may move with the user system 1110. When the two stationarycircles are aligned with the two circles that move with the user system1110, the image capture device and/or the user system 1110 may bealigned for the next image.

In some embodiments, after an image is taken by an image capture device,the image capture position module 1204 may take a sensor measurement ofthe position of the image capture device (e.g., including orientation,tilt, and the like). The image capture position module 1204 maydetermine one or more edges of the image that was taken by calculatingthe location of the edge of a field of view based on the sensormeasurement. Additionally, or alternatively, the image capture positionmodule 1204 may determine one or more edges of the image by scanning theimage taken by the image capture device, identifying objects within thatimage (e.g., using machine learning models discussed herein),determining one or more edges of the image, and positioning objects(e.g., circles or other shapes) at the edge of a display on the usersystem 1110.

The image capture position module 1204 may display two objects within adisplay of the user system 1110 that indicates the positioning of thefield of view for the next picture. These two objects may indicatepositions in the environment that represent where there is an edge ofthe last image. The image capture position module 1204 may continue toreceive sensor measurements of the position of the image capture deviceand calculate two additional objects in the field of view. The twoadditional objects may be the same width apart as the previous twoobjects. While the first two objects may represent an edge of the takenimage (e.g., the far right edge of the image), the next two additionalobjects representing an edge of the field of view may be on the oppositeedge (e.g., the far left edge of the field of view). By having the userphysically aligning the first two objects on the edge of the image withthe additional two objects on the opposite edge of the field of view,the image capture device may be positioned to take another image thatcan be more effectively stitched together without a tripod. This processcan continue for each image until the user determines the desiredpanorama has been captured.

Although multiple objects are discussed herein, it will be appreciatedthat the image capture position module 1204 may calculate the positionof one or more objects for positioning the image capture device. Theobjects may be any shape (e.g., circular, oblong, square, emoji, arrows,or the like). In some embodiments, the objects may be of differentshapes.

In some embodiments, there may be a distance between the objects thatrepresent the edge of a captured image and the distance between theobjects of a field of view. The user may be guided to move forward tomove away to enable there to be sufficient distance between the objects.Alternately, the size of the objects in the field of view may change tomatch a size of the objects that represent an edge of a captured imageas the image capture device approaches the correct position (e.g., bycoming closer or farther away from a position that will enable the nextimage to be taken in a position that will improve stitching of images.

In some embodiments, the image capture position module 1204 may utilizeobjects in an image captured by the image capture device to estimate theposition of the image capture device. For example, the image captureposition module 1204 may utilize GPS coordinates to determine thegeographical location associated with the image. The image captureposition module 1204 may use the position to identify landmarks that maybe captured by the image capture device.

The image capture position module 1204 may include a 2D machine learningmodel to convert 2D images into 2D panoramic images. The image captureposition module 1204 may include a 3D machine learning model to convert2D images to 3D representations. In one example, a 3D representation maybe utilized to display a three-dimensional walkthrough or visualizationof an interior and/or exterior environment.

The 2D machine learning model may be trained to stitch or assist institching two or more 2D images together to form a 2D panorama image.The 2D machine learning model may, for example, be a neural networktrained with 2D images that include physical objects in the images aswell as object identifying information to train the 2D machine learningmodel to identify objects in subsequent 2D images. The objects in the 2Dimages may assist in determining position(s) within a 2D image to assistin determining edges of the 2D image, warping in the 2D image, andassist in alignment of the image. Further, the objects in the 2D imagesmay assist in determining artifacts in the 2D image, blending of anartifact or border between two images, positions to cut images, and/orcrop the images.

In some embodiments, the 2D machine learning model may, for example, bea neural network trained with 2D images that include depth information(e.g., from a LiDAR device or structured light device of the user system1110 or the 3D and panoramic capture and stitching system 1102) of theenvironment as well as include physical objects in the images toidentify the physical objects, position of the physical objects, and/orposition of the image capture device/field of view. The 2D machinelearning model may identify physical objects as well as their depthrelative to other aspects of the 2D images to assist in the alignmentand position of two 2D images for stitching (or to stitch the two 2Dimages).

The 2D machine learning model may include any number of machine learningmodels (e.g., any number of models generated by neural networks or thelike).

The 2D machine learning model may be stored on the 3D and panoramiccapture and stitching system 1102, the image stitching and processorsystem 1106, and/or the user system 1110. In some embodiments, the 2Dmachine learning model may be trained by the image stitching andprocessor system 1106.

The image capture position module 1204 may estimate the position of theimage capture device (a position of the field of view of the imagecapture device) based on a seam between two or more 2D images from thestitching module 1206, the image warping from the cropping module 1208,and/or the graphical cut from the graphical cut module 1210.

The stitching module 1206 may combine two or more 2D images to generatea 2D panoramic. Based on the seam between two or more 2D images from thestitching module 1206, the image warping from the cropping module 1208,and/or a graphical cut, which has a field of view that is greater thanthe field of views of each of the two or more images.

The stitching module 1206 may be configured to align or “stitchtogether” two different 2D images providing different perspectives ofthe same environment to generate a panoramic 2D image of theenvironment. For example, the stitching module 1206 can employ known orderived (e.g., using techniques described herein) information regardingthe capture positions and orientations of respective 2D images to assistin stitching two images together.

The stitching module 1206 may receive two 2D images. The first 2D imagemay have been taken immediately before the second image or within apredetermined period of time. In various embodiments, the stitchingmodule 1206 may receive positioning information of the image capturedevice associated with the first image and then positioning informationassociated with the second image. The positioning information may beassociated with an image based on, at the time the image was taken,positioning data from the IMU, GPS, and/or information provided by theuser.

In some embodiments, the stitching module 1206 may utilize a 2D machinelearning module for scanning both images to recognize objects withinboth images, including objects (or parts of objects) that may be sharedby both images. For example, the stitching module 1206 may identify acorner, pattern on a wall, furniture, or the like shared at oppositeedges of both images.

The stitching module 1206 may align edges of the two 2D images based onthe positioning of the shared objects (or parts of objects), positioningdata from the IMU, positioning data from the GPS, and/or informationprovided by the user and then combine the two edges of the images (i.e.,“stitch” them together). In some embodiments, the stitching module 1206may identify a portion of the two 2D images that overlap each other andstitch the images at the position that is overlapped (e.g., using thepositioning data and/or the results of the 2D machine learning model.

In various embodiments, the 2D machine learning model may be trained touse the positioning data from the IMU, positioning data from the GPS,and/or information provided by the user to combine or stitch the twoedges of the images. In some embodiments, the 2D machine learning modelmay be trained to identify common objects in both 2D images to align andposition the 2D images and then combine or stitch the two edges of theimages. In further embodiments, the 2D machine learning model may betrained to use the positioning data and object recognition to align andposition the 2D images and then stitch the two edges of the imagestogether to form all or part of the panoramic 2D image.

The stitching module 1206 may utilize depth information for therespective images (e.g., pixels in the respective images, objects in therespective images, or the like) to facilitate aligning the respective 2Dimages to one another in association with generating a single 2Dpanoramic image of the environment.

The cropping module 1208 may resolve issues with two or more 2D imageswhere the image capture device was not held in the same position when 2Dimages were captured. For example, while capturing an image, the usermay position the user system 1110 in a vertical position. However, whilecapturing another image, the user may position the user system at anangle. The resultant images may not be aligned and may suffer fromparallax effects. Parallax effects may occur when foreground andbackground objects do not line up in the same way in the first image andthe second image.

The cropping module 1208 may utilize the 2D machine learning model (byapplying positioning information, depth information, and/or objectrecognition) to detect changes in the position of the image capturedevice in two or more images and then measure the amount of change inposition of the image capture device. The cropping module 1208 may warpone or multiple 2D images so that the images may be able to line uptogether to form a panoramic image when the images are stitched, andwhile at the same time preserving certain characteristics of the imagessuch as keeping a straight line straight.

The output of the cropping module 1208 may include the number of pixelcolumns and rows to offset each pixel of the image to straighten out theimage. The amount of offset for each image may be outputted in the formof a matrix representing the number of pixel columns and pixel rows tooffset each pixel of the image.

In some embodiments, the cropping module 1208 may determine the amountof image warping to perform on one or more of the multiple 2D imagescaptured by the image capture devices of the user system 1110 based onone or more image capture position from the image capture positionmodule 1204 or seam between two or more 2D images from the stitchingmodule 1206, the graphical cut from the graphical cut module 1210, orblending of colors from the blending module 1211.

The graphical cut module 1210 may determine where to cut or slice one ormore of the 2D images captured by the image capture device. For example,the graphical cut module 1210 may utilize the 2D machine learning modelto identify objects in both images and determine that they are the sameobject. The image capture position module 1204, the cropping module1208, and/or the graphical cut module 1210 may determine that the twoimages cannot be aligned, even if warped. The graphical cut module 1210may utilize the information from the 2D machine learning model toidentify sections of both images that may be stitched together (e.g., bycutting out a part of one or both images to assist in alignment andpositioning). In some embodiments, the two 2D images may overlap atleast a portion of the physical world represented in the images. Thegraphical cut module 1210 may identify an object, such as the samechair, in both images. However, the images of the chair may not line upto generate a panoramic that is not distorted and would not correctlyrepresent the portion of the physical world, even after image capturepositioning and image wrapping by the cropping module 1208. Thegraphical cut module 1210 may select one of the two images of the chairto be the correct representation (e.g., based on misalignment,positioning, and/or artifacts of one image when compared to the other)and cut the chair from the image with misaligning, errors inpositioning, and/or artifacts. The stitching module 1206 maysubsequently stitch the two images together.

The graphical cut module 1210 may try both combinations, for example,cutting the image of the chair from the first image and stitching thefirst image, minus the chair to the second image, to determine whichgraphical cut generates a more accurate panoramic image. The output ofthe graphical cut module 1210 may be a location to cut one or more ofthe multiple 2D images which correspond to the graphical cut, whichgenerates a more accurate panoramic image.

The graphical cut module 1210 may determine how to cut or slice one ormore of the 2D images captured by the image capture device based on oneor more image capture position from the image capture position module1204, stitching, or seam between two or more 2D images from thestitching module 1206, the image warping from the cropping module 1208,and the graphical cut from the graphical cut module 1210.

The blending module 1211 may colors at the seams (e.g., stitching)between two images so that the seams are invisible. Variation inlighting and shadows may cause the same object or surface to beoutputted in slightly different colors or shades. The blending modulemay determine the amount of color blending required based on one or moreimage capture position from the image capture position module 1204,stitching, image colors along the seams from both images, the imagewarping from the cropping module 1208, and/or the graphical cut from thegraphical cut module 1210.

In various embodiments, the blending module 1211 may receive a panoramafrom a combination of two 2D images and then sample colors along theseam of the two 2D images. The blending module 1211 may receive seamlocation information from the image capture position module 1204 toenable the blending module 1211 to sample colors along the seam anddetermine differences. If there is a significant difference in coloralong a seam between the two images (e.g., within a predeterminedthreshold of color, hue, brightness, saturation, and/or the like), theblending module 1211 may blend a predetermined size of both images alongthe seam at the position where there is the difference. In someembodiments, the greater the difference in color or image along theseam, the greater the amount of space along the seam of the two imagesthat may be blended.

In some embodiments, after blending, the blending module 1211 mayre-scan and sample colors along the seam to determine if there are otherdifferences in image or color that exceed the predetermined threshold ofcolor, hue, brightness, saturation, and/or the like. If so, the blendingmodule 1211 may identify the portions along the seam and continue toblend that portion of the image. The blending module 1211 may continueto resample the images along the seam until there are no furtherportions of the images to blend (e.g., any differences in color arebelow the predetermined threshold(s).)

The 3D image generator 1214 may receive 2D panoramic images and generate3D representations. In various embodiments, the 3D image generator 1214utilizes a 3D machine learning model to transform the 2D panoramicimages into 3D representations. The 3D machine learning model may betrained using 2D panoramic images and depth data (e.g., from a LiDARsensor or structured light device) to create 3D representations. The 3Drepresentations may be tested and reviewed for curation and feedback. Insome embodiments, the 3D machine learning model may be used with 2Dpanoramic images and depth data to generate the 3D representations.

In various embodiments, the accuracy, speed of rendering, and quality ofthe 3D representation generated by the 3D image generator 1214 aregreatly improved by utilizing the systems and methods described herein.For example, by rendering a 3D representation from 2D panoramic imagesthat have been aligned, positioned, and stitched using methods describedherein (e.g., by alignment and positioning information provided byhardware, by improved positioning caused by the guidance provided to theuser during image capture, by cropping and changing warping of images,by cutting images to avoid artifacts and overcome warping, by blendingimages, and/or any combination), the accuracy, speed of rendering, andquality of the 3D representation are improved. Further, it will beappreciated that by utilizing 2D panoramic images that have beenaligned, positioned, and stitched using methods described herein,training of the 3D machine learning model may be greatly improved (e.g.,in terms of speed and accuracy). Further, in some embodiments, the 3Dmachine learning model may be smaller and less complex because of thereduction of processing and learning that would have been used toovercome misalignments, errors in positioning, warping, poor graphiccutting, poor blending, artifacts, and the like to generate reasonablyaccurate 3D representations.

The trained 3D machine learning model may be stored in the 3D andpanoramic capture and stitching system 1102, image stitching andprocessor system 106, and/or the user system 1110.

In some embodiments, the 3D machine learning model may be trained usingmultiple 2D images and depth data from the image capture device of theuser system 1110 and/or the 3D and panoramic capture and stitchingsystem 1102. In addition, the 3D image generator 1214 may be trainedusing image capture position information associated with each of themultiple 2D images from the image capture position module 1204, seamlocations to align or stitch each of the multiple 2D images from thestitching module 1206, pixel offset(s) for each of the multiple 2Dimages from the cropping module 1208, and/or the graphical cut from thegraphical cut module 1210. In some embodiments, the 3D machine learningmodel may be used with 2D panoramic images, depth data, image captureposition information associated with each of the multiple 2D images fromthe image capture position module 1204, seam locations to align orstitch each of the multiple 2D images from the stitching module 1206,pixel offset(s) for each of the multiple 2D images from the croppingmodule 1208, and/or the graphical cut from the graphical cut module 1210to generate the 3D representations.

The stitching module 1206 may be a part of a 3D model that convertsmultiple 2D images into 2D panoramic or 3D panoramic images. In someembodiments, the 3D model is a machine learning algorithm, such as a3D-from-2D prediction neural network model. The cropping module 1208 maybe a part of a 3D model that converts multiple 2D images into 2Dpanoramic or 3D panoramic images. In some embodiments, the 3D model is amachine learning algorithm, such as a 3D-from-2D prediction neuralnetwork model. The graphical cut module 1210 may be a part of a 3D modelthat converts multiple 2D images into 2D panoramic or 3D panoramicimages. In some embodiments, the 3D model is a machine learningalgorithm, such as a 3D-from-2D prediction neural network model. Theblending module 1211 may be a part of a 3D machine learning model thatconverts multiple 2D images into 2D panoramic or 3D panoramic images. Insome embodiments, the 3D model is a machine learning algorithm, such asa 3D-from-2D prediction neural network model.

The 3D image generator 1214 may generate a weighting for each of theimage capture position module 1204, the cropping module 1208, thegraphical cut module 1210, and the blending module 1211, which mayrepresent the reliability or a “strength” or “weakness” of the module.In some embodiments, the sum of the weightings of the modules equals 1.

In cases where depth data is not available for the multiple 2D images,the 3D image generator 1214 may determine depth data for one or moreobjects in the multiple 2D images captured by the image capture deviceof the user system 1110. In some embodiments, the 3D image generator1214 may derive the depth data based on images captured by stereo-imagepairs. The 3D image generator can evaluate stereo image pairs todetermine data about the photometric match quality between the images atvarious depths (a more intermediate result), rather than determiningdepth data from a passive stereo algorithm.

The 3D image generator 1214 may be a part of a 3D model that convertsmultiple 2D images into 2D panoramic or 3D panoramic images. In someembodiments, the 3D model is a machine learning algorithm, such as a3D-from-2D prediction neural network model.

The captured 2D image datastore 1216 may be any structure and/orstructures suitable for captured images and/or depth data (e.g., anactive database, a relational database, a self-referential database, atable, a matrix, an array, a flat file, a documented-oriented storagesystem, a non-relational No-SQL system, an FTS-management system such asLucene/Solar, and/or the like). The captured 2D image datastore 1216 maystore images captured by the image capture device of the user system1110. In various embodiments, the captured 2D image datastore 1216stores depth data captured by one or more depth sensors of the usersystem 1110. In various embodiments, the captured 2D image datastore1216 stores image capture device parameters associated with the imagecapture device, or capture properties associated with each of themultiple image captures, or depth captures used to determine the 2Dpanoramic image. In some embodiments, the image datastore 1108 storespanoramic 2D panoramic images. The 2D panoramic images may be determinedby the 3D and panoramic capture and stitching system 1102 or the imagestitching and processor system 106. Image capture device parameters mayinclude lighting, color, image capture lens focal length, maximumaperture, angle of tilt, and the like. Capture properties may includepixel resolution, lens distortion, lighting, and other image metadata.

The 3D panoramic image datastore 1218 may be any structure and/orstructures suitable for 3D panoramic images (e.g., an active database, arelational database, a self-referential database, a table, a matrix, anarray, a flat file, a documented-oriented storage system, anon-relational No-SQL system, an FTS-management system such asLucene/Solar, and/or the like). The 3D panoramic image datastore 1218may store 3D panoramic images generated by the 3D and panoramic captureand stitching system 1102. In various embodiments, the 3D panoramicimage datastore 1218 stores properties associated with the image capturedevice or properties associated with each of the multiple image captureor depth capture used to determine the 3D panoramic image. In someembodiments, the 3D panoramic image datastore 1218 stores the 3Dpanoramic images. The 2D or 3D panoramic images may be determined by the3D and panoramic capture and stitching system 1102 or the imagestitching and processor system 106.

FIG. 13 depicts a flow chart 1300 of a 3D panoramic image capture andgeneration process according to some embodiments. In step 1302, theimage capture device may capture multiple 2D images using the imagesensor 920 and the WFOV lens 918 of FIG. 9 . The wider FOV means thatthe environment capture system 402 will require fewer scans to obtain a360° view. The WFOV lens 918 may also be wider horizontally as well asvertically. In some embodiments, the image sensor 920 captures RGBimages. In one embodiment, the image sensor 920 captures black and whiteimages.

In step 1304, the environment capture system may send the captured 2Dimages to the image stitching and processor system 1106. The imagestitching and processor system 1106 may apply a 3D modeling algorithm tothe captured 2D images to generate a panoramic 2D image. In someembodiments, the 3D modeling algorithm is a machine learning algorithmto stitch the captured 2D images into a panoramic 2D image. In someembodiments, step 1304 may be optional.

In step 1306, the LiDAR 912 and WFOV lens 918 of FIG. 9 may captureLiDAR data. The wider FOV means that the environment capture system 400will require fewer scans to obtain a 360° view.

In step 1308, the LiDAR data may be sent to the image stitching andprocessor system 1106. The image stitching and processor system 1106 mayinput the LiDAR data and the captured 2D image into the 3D modelingalgorithm to generate the 3D panoramic image. The 3D modeling algorithmis a machine learning algorithm.

In step 1310, the image stitching and processor system 1106 generatesthe 3D panoramic image. The 3D panoramic image may be stored in theimage datastore 408. In one embodiment, the 3D panoramic image generatedby the 3D modeling algorithm is stored in the image stitching andprocessor system 1106. In some embodiments, the 3D modeling algorithmmay generate a visual representation of the floorplan of the physicalenvironment as the environment capture system is utilized to capturevarious parts of the physical environment.

In step 1312, image stitching and processor system 1106 may provide atleast a portion of the generated 3D panoramic image to the user system1110. The image stitching and processor system 1106 may provide thevisual representation of the floorplan of the physical environment.

The order of one or more steps of the flow chart 1300 may be changedwithout affecting the end product of the 3D panoramic image. Forexample, the environment capture system may interleave image capturewith the image capture device with LiDAR data or depth informationcapture with the LiDAR 912. For example, the image capture device maycapture an image of section of the physical environment with the imagecapture device, and then LiDAR 912 obtains depth information fromsection 1605. Once the LiDAR 912 obtains depth information from section,the image capture device may move on to capture an image of anothersection, and then LiDAR 912 obtains depth information from section,thereby interleaving image capture and depth information capture.

In some embodiments, the devices and/or systems discussed herein employone image capture device to capture 2D input images. In someembodiments, the one or more image capture devices 1116 can represent asingle image capture device (or image capture lens). In accordance withsome of these embodiments, the user of the mobile device housing theimage capture device can be configured to rotate about an axis togenerate images at different capture orientations relative to theenvironment, wherein the collective fields of view of the images span upto 360° horizontally.

In various embodiments, the devices and/or systems discussed herein mayemploy two or more image capture devices to capture 2D input images. Insome embodiments, the two or more image capture devices can be arrangedin relative positions to one another on or within the same mobilehousing such that their collective fields of view span up to 360°. Insome embodiments, pairs of image capture devices can be used capable ofgenerating stereo-image pairs (e.g., with slightly offset yet partiallyoverlapping fields of view). For example, the user system 1110 (e.g.,the device the comprises the one or more image capture devices used tocapture the 2D input images) can comprise two image capture devices withhorizontal stereo offset fields of-view capable of capturing stereoimage pairs. In another example, the user system 1110 can comprise twoimage capture devices with vertical stereo offset fields-of-view capableof capturing vertical stereo image pairs. In accordance with either ofthese examples, each of the cameras can have fields-of-view that span upto 360. In this regard, in one embodiment, the user system 1110 canemploy two panoramic cameras with vertical stereo offsets capable ofcapturing pairs of panoramic images that form stereo pairs (withvertical stereo offsets).

The positioning component 1118 may include any hardware and/or softwareconfigured to capture user system position data and/or user systemlocation data. For example, the positioning component 1118 includes anIMU to generate the user system 1110 position data in association withthe one or more image capture devices of the user system 1110 used tocapture the multiple 2D images. The positioning component 1118 mayinclude a GPS unit to provide GPS coordinate information in associationwith the multiple 2D images captured by one or more image capturedevices. In some embodiments, the positioning component 1118 maycorrelate position data and location data of the user system withrespective images captured using the one or more image capture devicesof the user system 1110.

Various embodiments of the apparatus provide users with 3D panoramicimages of indoor as well as outdoor environments. In some embodiments,the apparatus may efficiently and quickly provide users with 3Dpanoramic images of indoor and outdoor environments using a single widefield-of-view (FOV) lens and a single light and detection and rangingsensors (LiDAR sensor).

The following is an example use case of an example apparatus describedherein. The following use case is of one of the embodiments. Differentembodiments of the apparatus, as discussed herein, may include one ormore similar features and capabilities as that of the use case.

FIG. 14 depicts a flow chart of a 3D and panoramic capture and stitchingprocess 1400 according to some embodiments. The flow chart of FIG. 14refers to the 3D and panoramic capture and stitching system 1102 asincluding the image capture device, but, in some embodiments, the datacapture device may be the user system 1110.

In step 1402, the 3D and panoramic capture and stitching system 1102 mayreceive multiple 2D images from at least one image capture device. Theimage capture device of the 3D and panoramic capture and stitchingsystem 1102 may be or include a complementary metal-oxide-semiconductor(CMOS) image sensor. In various embodiments, the image capture device isa charged coupled device (CCD). In one example, the image capture deviceis a red-green-blue (RGB) sensor. In one embodiment, the image capturedevice is an IR sensor. Each of the multiple 2D images may havepartially overlapping fields of view with at least one other image ofthe multiple 2D images. In some embodiments, at least some of themultiple 2D images combine to create a 360° view of the physicalenvironment (e.g., indoor, outdoor, or both).

In some embodiments, all of the multiple 2D images are received from thesame image capture device. In various embodiments, at least a portion ofthe multiple 2D images is received from two or more image capturedevices of the 3D and panoramic capture and stitching system 1102. Inone example, the multiple 2D images include a set of RGB images and aset of IR images, where the IR images provide depth data to the 3D andpanoramic capture and stitching system 1102. In some embodiments, each2D image may be associated with depth data provided from a LiDAR device.Each of the 2D images may, in some embodiments, be associated withpositioning data.

In step 1404, the 3D and panoramic capture and stitching system 1102 mayreceive capture parameters and image capture device parametersassociated with each of the received multiple 2D images. Image capturedevice parameters may include lighting, color, image capture lens focallength, maximum aperture, a field of view, and the like. Captureproperties may include pixel resolution, lens distortion, lighting, andother image metadata. The 3D and panoramic capture and stitching system1102 may also receive the positioning data and the depth data.

In step 1406, the 3D and panoramic capture and stitching system 1102 maytake the received information from steps 1402 and 1404 for stitching the2D images to form a 2D panoramic image. The process of stitching the 2Dimages is further discussed with regard to the flowchart of FIG. 15 .

In step 1408, the 3D and panoramic capture and stitching system 1102 mayapply a 3D machine learning model to generate a 3D representation. The3D representation may be stored in a 3D panoramic image datastore. Invarious embodiments, the 3D representation is generated by the imagestitching and processor system 1106 In some embodiments, the 3D machinelearning model may generate a visual representation of the floorplan ofthe physical environment as the environment capture system is utilizedto capture various parts of the physical environment.

In step 1410, the 3D and panoramic capture and stitching system 1102 mayprovide at least a portion of the generated 3D representation or modelto the user system 1110. The user system 1110 may provide the visualrepresentation of the floorplan of the physical environment.

In some embodiments, the user system 1110 may send the multiple 2Dimages, capture parameters, and image capture parameters to the imagestitching and processor system 1106. In various embodiments, the 3D andpanoramic capture and stitching system 1102 may send the multiple 2Dimages, capture parameters, and image capture parameters to the imagestitching and processor system 1106.

The image stitching and processor system 1106 may process the multiple2D images captured by the image capture device of the user system 1110and stitch them into a 2D panoramic image. The 2D panoramic imageprocessed by the image stitching and processor system 1106 may have ahigher pixel resolution than the 2D panoramic image obtained by the 3Dand panoramic capture and stitching system 1102.

In some embodiments, the image stitching and processor system 106 mayreceive the 3D representation and output a 3D panoramic image with pixelresolution that is higher than that of the received 3D panoramic image.The higher pixel resolution panoramic images may be provided to anoutput device with a higher screen resolution than the user system 1110,such as a computer screen, projector screen, and the like. In someembodiments, the higher pixel resolution panoramic images may provide tothe output device a panoramic image in greater detail and may bemagnified.

FIG. 15 depicts a flow chart showing further detail of one step of the3D and panoramic capture and stitching process of FIG. 14 . In step1502, the image capture position module 1204 may determine image capturedevice position data associated with each image captured by the imagecapture device. The image capture position module 1204 may utilize theIMU of the user system 1110 to determine the position data of the imagecapture device (or the field of view of the lens of the image capturedevice). The position data may include the direction, angle, or tilt ofone or more image capture devices when taking one or more 2D images. Oneor more of the cropping module 1208, the graphical cut module 1210, orthe blending module 1212 may utilize the direction, angle, or tiltassociated with each of the multiple 2D images to determine how to warp,cut, and/or blend the images.

In step 1504, the cropping module 1208 may warp one or more of themultiple 2D images so that two images may be able to line up together toform a panoramic image and while at the same time preserving specificcharacteristics of the images such as keeping a straight line straight.The output of the cropping module 1208 may include the number of pixelcolumns and rows to offset each pixel of the image to straighten out theimage. The amount of offset for each image may be outputted in the formof a matrix representing the number of pixel columns and pixel rows tooffset each pixel of the image. In this embodiment, the cropping module1208 may determine the amount of warping each of the multiple 2D imagesrequires based on the image capture pose estimation of each of themultiple 2D images.

In step 1506, the graphical cut module 1210 determines where to cut orslice one or more of the multiple 2D images. In this embodiment, thegraphical cut module 1210 may determine where to cut or slice each ofthe multiple 2D images based on the image capture pose estimation andthe image warping of each of the multiple 2D images.

In step 1508, the stitching module 1206 may stitch two or more imagestogether using the edges of the images and/or the cuts of the images.The stitching module 1206 may align and/or position images based onobjects detected within the images, warping, cutting of the image,and/or the like.

In step 1510, the blending module 1212 may adjust the color at the seams(e.g., stitching of two images) or the location on one image thattouches or connects to another image. The blending module 1212 maydetermine the amount of color blending required based on one or moreimage capture positions from the image capture position module 1204, theimage warping from the cropping module 1208, and the graphical cut fromthe graphical cut module 1210.

The order of one or more steps of the 3D and panoramic capture andstitching process 1400 may be changed without affecting the end productof the 3D panoramic image. For example, the environment capture systemmay interleave image capture with the image capture device with LiDARdata or depth information capture. For example, the image capture devicemay capture an image of a section 1605 of FIG. 16 of the physicalenvironment with the image capture device, and then LiDAR 612 obtainsdepth information from the section 1605. Once the LiDAR obtains depthinformation from the section 1605, the image capture device may move onto capture an image of another section 1610, and then LiDAR 612 obtainsdepth information from the section 1610, thereby interleaving imagecapture and depth information capture.

FIG. 16 depicts a block diagram of an example digital device 1602according to some embodiments. Any of the user system 1110, the 3Dpanoramic capture and stitching system 1102, and the image stitching andprocessor system may comprise an instance of the digital device 1602.Digital device 1602 comprises a processor 1604, a memory 1606, a storage1608, an input device 1610, a communication network interface 1612, anoutput device 1614, an image capture device 1616, and a positioningcomponent 1618. Processor 1604 is configured to execute executableinstructions (e.g., programs). In some embodiments, the processor 1604comprises circuitry or any processor capable of processing theexecutable instructions.

Memory 1606 stores data. Some examples of memory 1606 include storagedevices, such as RAM, ROM, RAM cache, virtual memory, etc. In variousembodiments, working data is stored within memory 1606. The data withinmemory 1606 may be cleared or ultimately transferred to storage 1608.

Storage 1608 includes any storage configured to retrieve and store data.Some examples of storage 1608 include flash drives, hard drives, opticaldrives, and/or magnetic tape. Each of memory 1606 and storage 1608comprises a computer-readable medium, which stores instructions orprograms executable by processor 1604.

The input device 1610 is any device that inputs data (e.g., touchkeyboard, stylus). Output device 1614 outputs data (e.g., speaker,display, virtual reality headset). It will be appreciated that storage1608, input device 1610, and an output device 1614. In some embodiments,the output device 1614 is optional. For example, routers/switchers maycomprise processor 1604 and memory 1606 as well as a device to receiveand output data (e.g., a communication network interface 1612 and/oroutput device 1614).

The communication network interface 1612 may be coupled to a network(e.g., communication network 104) via communication network interface1612. Communication network interface 1612 may support communicationover an Ethernet connection, a serial connection, a parallel connection,and/or an ATA connection. Communication network interface 1612 may alsosupport wireless communication (e.g., 802.16 a/b/g/n, WiMAX, LTE,Wi-Fi). It will be apparent that the communication network interface1612 may support many wired and wireless standards.

A component may be hardware or software. In some embodiments, thecomponent may configure one or more processors to perform functionsassociated with the component. Although different components arediscussed herein, it will be appreciated that the server system mayinclude any number of components performing any or all functionalitydiscussed herein.

The digital device 1602 may include one or more image capture devices1616. The one or more image capture devices 1616 can include, forexample, RGB cameras, HDR cameras, video cameras, and the like. The oneor more image capture devices 1616 can also include a video cameracapable of capturing video in accordance with some embodiments. In someembodiments, one or more image capture devices 1616 can include an imagecapture device that provides a relatively standard field-of-view (e.g.,around 75°). In other embodiments, the one or more image capture devices1616 can include cameras that provide a relatively wide field-of-view(e.g., from around 120° up to 360°), such as a fisheye camera, and thelike (e.g., the digital device 1602 may include or be included in theenvironment capture system 400).

A component may be hardware or software. In some embodiments, thecomponent may configure one or more processors to perform functionsassociated with the component. Although different components arediscussed herein, it will be appreciated that the server system mayinclude any number of components performing any or all functionalitydiscussed herein.

The invention claimed is:
 1. A method for an apparatus to gather imagesand depth information in a 360-degree scene, wherein the apparatuscomprises a first motor, an image capture device, and a depthinformation capture device, the method comprising: capturing, by theimage capture device while pointed in a first direction, a plurality ofimages at different exposures in a first field of view (FOV) of the360-degree scene; rotating, by the first motor, the depth informationcapture device and the image capture device about a first, substantiallyvertical, axis until the image capture device is pointed in a seconddirection, and capturing, by the depth information capture device whilebeing rotated, depth information for a first portion of the 360-degreescene; capturing, by the image capture device while pointed in thesecond direction, a plurality of images at different exposures in asecond FOV that overlaps the first FOV of the 360-degree scene;rotating, by the first motor, the depth information capture device andthe image capture device about the first axis until the image capturedevice is pointed in a third direction, and capturing, by the depthinformation capture device while being rotated, depth information for asecond portion of the 360-degree scene; and capturing, by the imagecapture device while pointed in the third direction, a plurality ofimages at different exposures in a third FOV that overlaps the secondFOV of the 360-degree scene.
 2. The method of claim 1, furthercomprising: rotating, by the first motor, the depth information capturedevice and the image capture device about the first axis until the imagecapture device is pointed in a fourth direction; and capturing, by theimage capture device while pointed in the fourth direction, a pluralityof images at different exposures in a fourth FOV that overlaps the firstand the third FOVs of the 360-degree scene.
 3. The method of claim 2,further comprising, while rotating, by the first motor, the depthinformation capture device and the image capture device about the firstaxis until the image capture device is pointed in the fourth direction,capturing, by the depth information capture device, depth informationfor a third portion of the 360-degree scene.
 4. The method of claim 3,wherein capturing, by the depth information capture device, depthinformation for the first, second and third portions of the 360-degreescene comprises capturing, by the depth information capture device,depth information for a first, second and third plurality of segments ofthe 360-degree scene.
 5. The method of claim 1, further comprising:stitching together, by an image stitching module, one or more of theplurality of images in the first, second and third FOVs into a panoramicimage of the 360-degree scene; and combining, by a 3D image generationmodule, the depth information for the first, second and third portionsof the 360-degree scene with the panoramic image of the 360-degree sceneto create a 3D panoramic image of the 360-degree scene.
 6. The method ofclaim 5, further comprising blending, by a blending module, two or moreof the plurality of images of the first, second and third FOVs intofirst, second and third blended images of the first, second and thirdFOVs, wherein stitching together, by the image stitching module, one ormore of the plurality of images in the first, second and third FOVs intothe panoramic image of the 360-degree scene comprises stitching togetherthe first, second and third blended images into the panoramic image ofthe 360-degree scene.
 7. The method of claim 6, wherein blending, by theblending module, two or more of the plurality of images of the first,second and third FOVs into the first, second and third blended images ofthe first, second and third FOVs comprises blending two or more of theplurality of images of the first, second and third FOVs into first,second and third high dynamic range (HDR) images of the first, secondand third FOVs, and wherein stitching together, by the image stitchingmodule, one or more of the plurality of images in the first, second andthird FOVs into the panoramic image of the 360-degree scene comprisesstitching together the first, second and third HDR images in the first,second and third FOVs into the panoramic image of the 360-degree scene.8. The method of claim 6, wherein the apparatus further comprises asecond motor, wherein the depth information capture device comprises aLight Detection and Ranging (LiDAR) device and a mirror, and the methodfurther comprises rotating, by the second motor, the mirror about asecond, substantially horizontal, axis, and emitting, by the LiDARdevice, a plurality of laser pulses to the mirror, the mirror, in turn,emitting the plurality of laser pulses in one or more revolutions aboutthe second axis and receiving a corresponding plurality of reflectedlaser pulses, and the LiDAR device, in turn, receiving the plurality ofreflected laser pulses and generating the depth information therefrom,wherein the first, second and third blended images of the first, secondand third FOVs each comprise a plurality of pixels, wherein each of theplurality of pixels is associated with numerical coordinates thatidentify a location of the pixel in the scene, and wherein each of thereflected laser pulses is likewise associated with correspondingnumerical coordinates that identify a location of the depth informationgenerated therefrom, and combining, by the 3D image generation module,the depth information for the 360-degree scene with the panoramic imageof the 360-degree scene to create the 3D panoramic image of the360-degree scene comprises combining the depth information at a locationin the 360-degree scene with the panoramic image at the location in the360-degree scene to create the 3D panoramic image of the 360-degreescene.
 9. The method of claim 1, wherein the image capture devicecomprises a lens situated at a no-parallax point substantially at acenter of the first axis such that as the first motor rotates the imagecapture device about the first axis to capture while the image capturedevice is pointed in the first, second, and third directions theplurality of images at different exposures in the respective first,second and third overlapping FOVs, a parallax effect between theplurality of images in the first, second and third overlapping FOVs isreduced or eliminated.
 10. The method of claim 1, wherein the apparatusfurther comprises a second motor, the depth information capture devicecomprises a Light Detection and Ranging (LiDAR) device and a mirror, andthe method further comprises rotating, by the second motor, the mirrorabout a second, substantially horizontal, axis, and emitting, by theLiDAR device, a plurality of laser pulses to the mirror, the mirror, inturn, emitting the plurality of laser pulses in one or more revolutionsabout the second axis and receiving a corresponding plurality ofreflected laser pulses, and the LiDAR device, in turn, receiving theplurality of reflected laser pulses and generating the depth informationtherefrom, so that as the image capture device and the LiDAR device arerotated by the first motor about the first axis and the mirror isrotated by the second motor about the second axis, the LiDAR devicecaptures the depth information in three dimensions for the 360-degreescene.
 11. An apparatus to gather images and depth information for usein a 360-degree scene, comprising: a first motor; an image capturedevice to capture, while pointed in a first direction, a plurality ofimages at different exposures in a first field of view (FOV) of the360-degree scene; and a depth information capture device, wherein thedepth information capture device and the image capture device arerotated by the first motor about a first, substantially vertical, axisuntil the image capture device is pointed in a second direction, thedepth information capture device while being rotated, to capture depthinformation for a first portion of the 360-degree scene, the imagecapture device to capture, while pointed in the second direction, aplurality of images at different exposures in a second FOV that overlapsthe first FOV of the 360-degree scene, the depth information capturedevice and the image capture device are rotated by the first motor aboutthe first axis until the image capture device is pointed in a thirddirection, the depth information capture device while being rotated, tocapture depth information for a portion of the 360-degree scene, and theimage capture device, while pointed in the third direction, to capture aplurality of images at different exposures in a third FOV that overlapsthe second FOV of the 360-degree scene.
 12. The apparatus of claim 11,wherein the depth information capture device and the image capturedevice are rotated by the first motor about the first axis until theimage capture device is pointed in a fourth direction, and the imagecapture device, while pointed in the fourth direction, to capture aplurality of images at different exposures in a fourth FOV that overlapsthe first and the third FOVs of the 360-degree scene.
 13. The apparatusof claim 12, wherein the depth information capture device, while beingrotated by the first motor about the first axis until the image capturedevice is pointed in the fourth direction, to capture depth informationfor a third portion of the 360-degree scene.
 14. The apparatus of claim13, wherein the depth information capture device while being rotated, tocapture depth information for the first, second and third portions ofthe 360-degree scene comprises the depth information capture devicewhile being rotated, to capture depth information for a first, secondand third plurality of segments of the 360-degree scene.
 15. Theapparatus of claim 11, further comprising: an image stitching module tostitch together one or more of the plurality of images in the first,second and third FOVs into a panoramic image of the 360-degree scene;and a 3D image generation module to combine the depth information forthe first, second and third portions of the 360-degree scene with thepanoramic image of the 360-degree scene to create a 3D panoramic imageof the 360-degree scene.
 16. The apparatus of claim 15, furthercomprising: a blending module to blend two or more of the plurality ofimages of the first, second and third FOVs into first, second and thirdblended images of the first, second and third FOVs, wherein the imagestitching module to stitch together one or more of the plurality ofimages in the first, second and third FOVs into the panoramic image ofthe 360-degree scene comprises the image stitching module stitchingtogether the first, second and third blended images into the panoramicimage of the 360-degree scene.
 17. The apparatus of claim 16, whereinthe blending module blending two or more of the plurality of images ofthe first, second and third FOVs into the first, second and thirdblended images of the first, second and third FOVs comprises theblending module blending two or more of the plurality of images of thefirst, second and third FOVs into first, second and third high dynamicrange (HDR) images of the first, second and third FOVs, and the imagestitching module stitching together the one or more of the plurality ofimages in the first, second and third FOVs into the panoramic image ofthe 360-degree scene comprises the image stitching module stitchingtogether the first, second and third HDR images in the first, second andthird FOVs into the panoramic image of the 360-degree scene.
 18. Theapparatus of claim 16, further comprising a second motor, wherein thedepth information capture device comprises a Light Detection and Ranging(LiDAR) device and a mirror, the second motor to rotate the mirror abouta second, substantially horizontal, axis while the LiDAR device emits aplurality of laser pulses to the mirror, the mirror, in turn, emits theplurality of laser pulses in one or more revolutions about the secondaxis and receives a corresponding plurality of reflected laser pulses,and the LiDAR device, in turn, receives the plurality of reflected laserpulses and generates the depth information therefrom, wherein the first,second and third blended images of the first, second and third FOVs eachcomprise a plurality of pixels, wherein each of the plurality of pixelsis associated with numerical coordinates that identify a location of thepixel in the scene, and wherein each of the reflected laser pulses islikewise associated with corresponding numerical coordinates thatidentify a location of the depth information generated therefrom, andcombining, by the 3D image generation module, the depth information forthe 360-degree scene with the panoramic image of the 360-degree scene tocreate the 3D panoramic image of the 360-degree scene comprisescombining the depth information at a location in the 360-degree scenewith the panoramic image at the location in the 360-degree scene tocreate the 3D panoramic image of the 360-degree scene.
 19. The apparatusof claim 11, wherein the image capture device comprises a lens situatedat a no-parallax point substantially at a center of the first axis suchthat as the first motor rotates the image capture device about the firstaxis to capture while the image capture device is pointed in the first,second, and third directions the plurality of images at differentexposures in the respective first, second and third overlapping FOVs, aparallax effect between the plurality of images in the first, second andthird overlapping FOVs is reduced or eliminated.
 20. The apparatus ofclaim 11, further comprising a second motor, wherein the depthinformation capture device comprises a Light Detection and Ranging(LiDAR) device and a mirror, the mirror rotated by the second motorabout a second, substantially horizontal, axis while the LiDAR deviceemits a plurality of laser pulses to the mirror, the mirror, in turn,emits the plurality of laser pulses in one or more revolutions about thesecond axis and receives a corresponding plurality of reflected laserpulses, the LiDAR device, in turn, receives the plurality of reflectedlaser pulses and generates the depth information therefrom, so that asthe image capture device and the LiDAR device are rotated by the firstmotor about the first axis and the mirror is rotated by the second motorabout the second axis, the LiDAR device captures the depth informationin three dimensions for the 360-degree scene.